Robot automated mining

ABSTRACT

In one embodiment, the present disclosure provides a robot automated mining method. In one embodiment, a method includes a robot positioning a charging component for entry into a drill hole. In one embodiment, a method includes a robot moving a charging component within a drill hole. In one embodiment, a method includes a robot filling a drill hole with explosive material. In one embodiment, a method includes operating a robot within a mining environment.

FIELD OF THE DISCLOSURE

This disclosure relates to mining environments and particularly tooperation of a robot in a mining environment.

BACKGROUND

A mining environment can include generally a mining wall having aplurality of drill holes. In one embodiment, a mining wall can include arock formation. The plurality of drill holes can extend a depth into themining wall. A mining wall may not be flat in a plane parallel to an x-yreference plane but instead may include a plurality of protrusions orcavities extending in a z direction parallel to a z axis. For removingmaterial of the mining wall, explosive material can be placed into theplurality of drill holes. The explosive material can be detonated forbreaking apart a portion of the mining wall.

SUMMARY

In one embodiment, the present disclosure provides a robot automatedmining method. In one embodiment, a method includes a robot positioninga charging component for entry into a drill hole. In one embodiment, amethod includes a robot moving a charging component within a drill hole.In one embodiment, a method includes a robot filling a drill hole withexplosive material. In one embodiment, a method includes operating arobot within a mining environment.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects of the present disclosure are particularly pointed outand distinctly claimed as examples in the claims at the conclusion ofthe specification. The foregoing and other objects, features, andadvantages of the disclosure are apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 is a schematic diagram illustrating a mining environment in whicha robot can operate in one embodiment according to the presentdisclosure;

FIG. 2 is a representation of a drilling map in one embodiment;

FIG. 3 is a schematic diagram illustrating a scan path within a miningenvironment in one embodiment;

FIG. 4 is a schematic block diagram illustrating a system having a robotin one embodiment;

FIG. 5 is a schematic diagram illustrating a tool assembly for use inmining in one embodiment;

FIG. 6 is a perspective physical form view illustrating a tool assemblyfor use in mining in one embodiment;

FIG. 7 is a perspective physical form view illustrating the toolassembly of FIG. 6 for use in mining in one embodiment;

FIG. 8 is a perspective view of a magazine for holding a detonatorpackage in one embodiment;

FIG. 9 is an operational perspective physical form view illustrating arobot in mining environment picking up a detonator package from themagazine of FIG. 8 in one embodiment;

FIG. 10 is an operational perspective physical form view illustrating arobot in mining environment performing visual servoing for positioning acharging component for drill hole entry in one embodiment;

FIG. 11A is a schematic side view illustrating a drill hole with acharging hose and a detonator package disposed therein in oneembodiment;

FIG. 11B is a schematic side view illustrating a drill hole with acharging hose in a state of being retracted by a robot in oneembodiment;

FIG. 12 is a flow diagram illustrating a method for charging in oneembodiment;

FIG. 13 is a flow diagram illustrating a method for charging in oneembodiment;

FIG. 14 is a flow diagram illustrating a method for charging in oneembodiment;

FIG. 15 is a flow diagram illustrating a method for charging in oneembodiment;

FIG. 16 is a flow diagram illustrating a method for positioning in oneembodiment;

FIG. 17 is a flow diagram illustrating a method for registering in oneembodiment;

FIG. 18 is a flow diagram illustrating a method for charging in oneembodiment;

FIG. 19 is a flow diagram illustrating a method for operating a robot ina mining environment in one embodiment;

FIG. 20 is a physical form view illustrating a robot user interface inone embodiment;

FIG. 21 is a schematic diagram illustrating a robot user interface inone embodiment;

FIG. 22 is a schematic diagram illustrating a robot user interface inone embodiment;

FIG. 23 is an operational perspective physical form view illustratingfirst and second robots working simultaneously in a mining environmentin one embodiment;

FIG. 24 is a schematic block diagram illustrating a computer system inone embodiment; and

FIG. 25 is a diagram illustrating a computer readable medium in oneembodiment.

DETAILED DESCRIPTION

The present disclosure addresses and enhances, inter alia, robot systemsand a method of performing mining operations. A method herein canincorporate a robot system having a robot.

A mining environment 500 is illustrated generally at FIG. 1. Miningenvironment 500 can have a mining wall 16 characterized by a pluralityof drill holes 18 (only two of which are shown in FIG. 1). Miningenvironment 500 can further include one or more system 100 having arobot 102. In one embodiment, robot 102 can include a base 102B, links102L, joints 102J, and a tool assembly 102T. Robot 102 may be supportedon a carrier 14. Tool assembly 102T can be connected to an end of arobot arm defined by links 102L and joints 102J. In one embodiment,robot 102 can include on board control features as well as externalcontrol features 101. For example, external control features 101 mayinclude a robot controller 150, operator computer system 170, and abackend computer system 160. Positions within mining environment 500 canbe defined with respect to a reference coordinate system havingreference x, y, and z axes as illustrated in FIG. 1. Additional figuresherein depict reference coordinate systems and positions withinenvironments illustrated by such additional figures can be defined withrespect to reference coordinate systems depicted. Referring to thereference z axis of the mining environment 500 of FIG. 1, drill holes 18can have respective axes z₁, z₂ that are substantively parallel toreference axis z.

Drill holes 18 may be previously formed using drilling equipment, e.g.,a drill rig. Drill holes 18 can be formed at locations according tolocations that are determined by blast engineers. At a drill holeformation stage, as shown in FIG. 2 along with FIG. 1, a drilling map116 can be provided that includes data defining a plan for blasting ofmining wall 16 represented by mining wall representation 16R in drillingmap 116. Drilling map 116 can have recorded therein coordinates forlocations of respective drill holes per a plan designed by a blastengineer and then, with use of drilling equipment drill holes 18represented by drill hole representations 18R in drilling map 116 can beformed at the general location of the coordinates. In this way, drillingmap 116 in one stage of its development can have information of thegeneral location of the drill holes 18 but not a precise location ofdrill holes 18 because the drill holes 18 may not be formed precisely atthe specified locations.

Drilling map 116 can be provided using, e.g., a combination of manualinputs and/or inputs from one or more sensor that senses surfacegeometry features of mining wall 16 represented by mining wallrepresentation 16R in drilling map 116. In one embodiment, drilling map116 can be provided in an intermediary form having surface geometryinformation. A blast engineer can examine the surface geometryinformation of drilling map 116 and can designate in drilling map 116coordinate locations for drill holes 18 represented by drill holerepresentations 18R in drilling map 116. Drilling map 116 can beprovided by a formatted computer file, e.g., in CAD format stored in amemory location of system 100 and can include a representation of thecoordinates and dimensions of drill holes 18. The coordinates anddimensions can be expressed in two dimensions and/or three dimensions.In addition to having coordinate and dimension information of drillholes 18, drilling map 116 can include an index of drill holes 18represented by drill hole representations 18R in drilling map 116. Forexample, each drill hole 18 represented by drill hole representations18R in drilling map 116 can have an index identifier that can bedesignated by a blast engineer. There can be associated with anidentifier for each drill hole 18 an identifier of a detonator package510. In forming drilling map 116, a blast engineer can select a bestdetonator package 510 to associate with each drill hole 18 and canassociate information identifying detonator package 510 to each drillhole 18, all of which information can be included in drilling map 116.In alternative embodiments, drilling map 116 can include a fewer numberof information items or a larger number of information items than theinformation items described.

Referring further to FIG. 1, FIG. 1 illustrates one embodiment of miningenvironment 500 having system 100 according to an embodiment of thepresent disclosure for imaging an environment. In one embodiment, system100 can be disposed within an underground area 12. System 100 can beregarded as a robot system or an imaging system. In this exemplaryembodiment, system 100 may be used in the charging of a plurality ofdrill holes 18 in mining. For example, system 100 may be employed inunderground mining and may generally include robot 102 having a camerasystem 104 (imager system). Camera system 104 can be included as part oftool assembly 102T and can be used for imaging mining wall 16 havingplurality of drill holes 18. System 100 can be operative to obtain oneor more image representation using camera system 104. In one embodimentthe one of more image representation can include 3-dimensional (3D)point cloud image data. Camera system 104 can provide imaging functionsenabling the robot to ‘see’ physical objects in its environment. Camerasystem 104 may be realized by proprietary and/or application-specificimaging device(s) or commercial off-the-shelf (COTS) offerings providing2-dimensional (2D), 3-dimensional (3D), and/or depth-sensing imagingcapabilities. An example COTS product is the Kinect® motion controlleroffered by Microsoft Corporation.

In one embodiment, system 100 can include magazine 15 (also shown inFIG. 9 and described in greater detail herein) for holding variousdetonator packages 510. In one embodiment, magazine 15 can be configuredto assemble detonator packages 510 (also shown in FIG. 8 and describedin greater detail herein). While reference to system 100 is made in thecontext of robotic or automated charging of drill holes 18 in mining, itwill be readily appreciated by those skilled in the art that thetechnique of the present disclosure is applicable to other situationswhere imaging is needed of an environment with one or more features.

FIG. 3 illustrates a three dimensional view of mining environment 500.With reference to FIGS. 1 and 3, mining environment 500 can includesystem 100 (FIG. 1) having robot 102 (FIG. 1), mining wall 16 havingdrill holes 18 (FIG. 1), a floor 22, a ceiling 23 and sidewalls 24 (FIG.3).

As shown in FIG. 4, system 100 can generally include robot 102 operablyconnected to robot controller 150, operator computer system 170, andbackend computer system 160, which are operably coupled viacommunication links 140. Robot controller 150, operator computer system170, and a backend computer system 160 can define external controlfeatures 101 of system 100. Components of external control features 101are depicted at an elevation in underground area 12 and can be disposedat any location above ground or below ground and at any distance fromrobot 102. Components of external control features 101 can be located onsite at a mining location and/or off site at a remote location. Operatorcomputer system 170 can include one or more user interface through whichone or more operator can input operator control that can be obtained forprocessing by system 100. Operator computer system 170 can permitteleoperation by one or more operator such as one or more local operatorand/or one or more remote operator. Operator computer system 170 can belocated locally to permit local teleoperation by a local operator and/orremotely to permit teleoperation by a remote operator.

Operator computer system 170, in one embodiment as indicated in FIG. 1,can include local operator computer system and supervisor operatorcomputer system and be distributed in both local and remote locations topermit operation by both one or more operator at a local location andone or more operator at a remote location. Robot 102 can include camerasystem 104, actuator system 110 (FIG. 4), vision assistance system 106(FIG. 4), and other sensor/sensor devices 108 (FIG. 4). In oneembodiment, camera system 104, actuator system 110, vision assistancesystem 106 (FIG. 4), and other sensor/sensing devices 108 (FIG. 4) canbe included as part of tool assembly 102T. Robot 102, robot controller150, operator computer system 170, and a backend computer system 160,each can have one or more computer system 800, (exemplary generalfeatures of which are described herein with reference to FIG. 24herein).

In one embodiment, as illustrated by the enlarged view of FIG. 1, toolassembly 102T can be configured to hold a charging hose 202. Toolassembly 102T can be configured to move charging hose 202 forwardly andoppositely backwardly along longitudinal axis 204 of charging hose 202as illustrated by double-headed arrow 206. Tool assembly 102T can beconfigured to rotate charging hose 202 in a first direction and a secondopposite direction about longitudinal axis 204 of charging hose 202 asillustrated by curved double-headed arrow 208. Tool assembly 102T caninclude camera system 104 and can be disposed on a robot arm defined bylinks 102L and joints 102J, e.g., at an end of a robot arm. Thecomponents and operation of system 100 are described in greater detailherein.

Further aspects of a tool assembly 102T in one embodiment areillustrated in FIG. 5 which provides a schematic view of tool assembly102T. Tool assembly 102T can include first and second wheel actuators214 for providing back and forth movement indicated by double-headedarrow 206 along axis of charging hose 202. Tool assembly 102T caninclude a chain actuator 220 for providing back and forth rotationalmovement of the charging hose 202 indicated by curved double-headedarrow 208 about longitudinal axis 204.

A perspective physical form view of a tool assembly 102T is shown inFIG. 6. For example tool assembly 102T can include frame assembly 230for supporting various components of tool assembly 102T. Tool assembly102T can include sleeve 232 for guiding charging hose 202. Tool assembly102T can include a plurality of wheel actuators 214 for providing backand forth movement of charging hose 202 along double-headed arrow 206 ina direction along longitudinal axis 204 of charging hose 202. A toolassembly 102T can also include camera system 104 disposed in a positionwith a view forward of a distal end 234 of tool assembly 102T.

FIG. 7 illustrates a rear perspective view of a tool assembly 102T. Toolassembly 102T can include wheel actuators 214 for providing back andforth movement of charging hose 202 along double-headed arrow 206 in adirection along longitudinal axis 204 of charging hose 202. Toolassembly 102T can include chain actuator 220 for providing movement ofcharging hose 202 along curved double-headed arrow 208 rotationally backand forth about longitudinal axis of charging hose 202.

FIGS. 8 and 9 further illustrate magazine 15 for holding a plurality ofdetonator packages 510. As shown in FIG. 8 each detonator package 510can include detonator primer 512, detonator 514, and signal wire 516.Magazine 15 can include a first area 522 for holding detonators 514(with respective signal wires 516 preattached thereto) and a second area523 for holding primers 512. Detonators 514 with extending signal wires516 and primers 512 can be placed on magazine 15 separately and magazine15 can be operative so that when magazine 15 is appropriately actuated,magazine 15 can assemble one or more primer 512 to one or more detonator514 at second area 523.

FIG. 9 illustrates robot 102 having tool assembly 102T in use picking upa detonator package 510. For picking up a detonator package 510 a distalend of charging hose 202 can extend from a distal end of tool assembly102T. Charging hose 202 can be configured to be friction fit about aprimer 512 of a detonator package 510. Tool assembly 102T can be movedto be friction fit about a primer 512 to thereby pick up detonatorpackage 510 having primer 512. Referring again to FIG. 8, magazine 15can include orientation feature 527. Robot 102 with tool assembly 102Tcan be operated in an automatic mode of operation or teleoperation modeto be disposed about feature 527 having coordinates that are fixed andknown in relation to remaining locations of magazine 15. Accordingly, bybeing operated so that tool assembly 102T is disposed about feature 527precise coordinate information as to all locations of system 100 can berecorded by system 100.

FIG. 10 illustrates tool assembly 102T in use positioning a detonatorpackage 510 for entry (insertion) into a drill hole 18, e.g., at aposition adjacent an opening 218 of a drill hole 18. FIG. 10 illustratesthat charging hose 202 held by tool assembly 102T can hold primer 512 tothereby hold detonator package 510, the detonator package 510 havingprimer 512, detonator 514 and signal wire 516. When charging hose 202with robot 102 is moved toward a distal end 318 of drill hole 18, primer512 of detonator package 510 can be moved to a distal end 318 of drillhole 18 and a signal wire 516 of detonator package 510 as shown canextend backwardly and generally coextensively with charging hose 202(generally running adjacent to an exterior of charging hose 202) so thatsignal wire 516 can be accessible from a location externally from drillhole 18 as shown in FIGS. 11A and 11B. FIG. 11B illustrates charginghose 202 being retracted from drill hole 18 by robot 102 in a directionindicated by arrow 231. As charging hose 202 is being retracted,explosive material 226 can be pumped in through charging hose 202 sothat when charging hose 202 is retracted, there is left within drillhole 18 a deposit of explosive material 226 which can be later detonatedusing detonator package 510. In one embodiment, a distal end 310 ofdrill hole 18 can refer to an areas of drill hole 18 spaced apart froman opening 218 of drill hole 18. In one embodiment a distal end 318 ofdrill hole 18 can be indicated by a threshold distance from an opening218 of drill hole 18 specified by a blast engineer and recorded indrilling map 116. In one embodiment a distal end of drill hole 18 can beindicated by a threshold distance from a drill hole opening 218, thethreshold distance regarded as sufficient for performance of blasting bya blast engineer and/or an operator of system 100.

With reference again to FIG. 1, system 100 can have an automated mode ofoperation in which robot 102 with tool assembly 102T automaticallyperforms a charging component positioning procedure for controlling apositioning of the charging component. A charging component as set forthherein in one embodiment can include one or more charging component,e.g. one or more of charging hose 202, detonator package 510, one ormore component of a detonator package 510 (e.g. primer 512, detonator514, and or/signal wire 516) and/or explosive material 226. System 100can have a teleoperation mode of operation in which robot 102 with toolassembly 102T performs the charging component positioning procedure forcontrolling the positioning of the charging component based on one ormore operator input. A charging component positioning procedure can be aprocedure for positioning a charging component for entry into drill hole18. A charging component positioning procedure can be a procedure formoving a charging component from an opening 218 of drill hole 18 to adistal end of drill hole 18 inclusive or any obstruction avoidancecomponent positioning routine performed for moving a charging componentfrom an opening 218 of drill hole 18 to a distal end of drill hole 18. Acharging component positioning procedure can be a procedure forovercoming an obstruction within drill hole 18. In one embodiment, for atime that an automated mode of operation is active, functions performedby system 100 can be independent of any current input of an operator. Inone embodiment, for the time that an automated mode of operation isactive, robot 102 with tool assembly 102T can move the chargingcomponent to perform the charging component positioning procedureindependent of any current input of an operator.

In one embodiment, system 100 can be configured so that system 100transitions from an automated mode to a teleoperation mode based on asensed condition sensed by system 100 e.g. using one or more imagerepresentation obtained using camera system 104. In one embodiment,system 100 can have a manual override feature and can be configured sothat system 100 transitions from an automated mode to a teleoperationmode based on an input of an operator entered using a user interface ofoperator computer system 170. As has been noted operator computer system170, in one embodiment, can include local operator computer system andsupervisor operator computer system and be distributed in both local andremote locations to permit operation by both one or more operator at alocal location and one or more operator at a remote location. In oneembodiment, system 100 can be configured so that the remote supervisoroperator computer system has priority over the local operator computersystem. In one embodiment, system 100 can be configured so that system100 disables a local operator computer system based on one or moreoperator input entered into a user interface of a remote supervisoroperator computer system of operator computer system 170.

One or more computer system of system 100, e.g., one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus (e.g., one or moreprocessor of one or more computer), cause the apparatus to perform theactions such as the methods shown in FIGS. 12-19 as described in greaterdetail herein.

Referring to the flow diagram of FIG. 12, one general aspect can includea method 900 for charging one or more drill hole 18 in mining wall 16.

In reference to the method of FIG. 12, there is set forth herein in oneembodiment a method 900 comprising at block 902 placing a robot 102having a tool assembly 102T that includes a camera system 104 at aposition in front of a mining wall 16, the robot 102 obtaining at block906 one or more image representation using the camera system 104,positioning at block 910 one or more charging component for entry intodrill hole 18, moving at block 914 with the robot 102 the one or morecharging component within the drill hole 18, and feeding at block 918with the robot 102 explosive material into the drill hole 18. The movingat block 914 can include in one embodiment inserting with the robot 102one or more charging component into a drill hole 18. The moving at block914 can include in one embodiment moving with the robot 102 one or morecharging component from an opening 218 of a drill hole 18 to a distalend 318 (FIG. 11B) of a drill hole 18. The moving at block 914 caninclude in one embodiment moving with the robot 102 one or more chargingcomponent to a distal end 318 (FIG. 11B) of a drill hole 18.

In one embodiment the one or more image representation obtained at block906 can include 3D point cloud image data. The obtaining of image dataat block 906 in one embodiment can include performing a prescan toobtain one or more image representation of a mining wall to determinemajor boundaries of a mining wall, performing a scan to obtain one ormore image representation for use in determining a scan path andperforming visual servoing. On performing visual servoing, a robot 102can obtain one or more image representation and holding the one or morecharging component can use the one or more image representation obtainedwith a robot camera system 104 to precisely locate the one or morecharging component. One or more image representation can represent amining wall, e.g. an entire mining wall 16 or a portion of a mining wall16 such as a portion having a drill hole 18. An image representation canbe obtained using camera system 104 in one embodiment in response tooperating camera system 104 for capture of an image representation. Animage representation can be obtained using camera system 104 in oneembodiment by merging image data of a plurality of obtained imagerepresentations. In one embodiment, for obtaining an imagerepresentation using camera system 104 system 100 can merge togetherimage data from a plurality of image representations representing theplurality of portions of mining wall 16. In one embodiment, forobtaining an image representation using camera system 104 system 100 canmerge together image data of a plurality of image representations whereone or more of the plurality of image representations can be obtained bysystem 100 e.g. in response to operating camera system 104 for captureof an image representation.

A method as set forth in reference to the flow diagram of FIG. 12 in oneembodiment can use drilling map 116. For example, in the performance ofa scan, pre-scan or visual servoing which can be performed as part ofthe robot obtaining image data at block 906 can use coordinates of thedrilling map 116 to determine an initial area of interest on a miningwall 16 on which to locate a field of view of camera system 104. Inanother aspect, system 100 can process obtained image data to detectdrill holes 18 which drill holes 18 if drilled according to a drillingmap 116 should have corresponding drill hole representations 18R (FIG.2) identified on drilling map 116. In one aspect a system 100 canregister detected drill holes 18 detected by processing of image datacaptured using camera system 104. In one aspect when a system 100detects a drill hole 18 represented in obtained image data system 100can perform registration of a drill hole 18 by associating a flag todrill hole identifier of drilling map 116 so that the drill hole 18identified by the identifier is designated as being detected in obtainedimage data. System 100 can use a pattern recognition process to detect adrill hole 18 and/or other features represented in one or more imagerepresentation obtained using camera system 104. The pattern recognitionprocess can employ one or more pattern recognition technology e.g. oneor more of clustering algorithms, ensemble learning algorithms,classification algorithms, sequence labeling algorithms, regressionalgorithms, or multilinear subspace learning algorithms.

In one embodiment a method as set forth in reference to the flow diagramof FIG. 12 can include use of a magazine 15 features of which aredescribed in one embodiment in reference to FIGS. 1, 8 and 9. In oneembodiment, the magazine 15 can hold one or more charging component,e.g. detonator package 510 and/or one or more component of a detonatorpackage 510 such as a primer 512, a detonator 514 or a signal wire 516.In one embodiment, a magazine 15 can be adapted to assemble componentsof a detonator package 510. In one embodiment a method according to theflow diagram of FIG. 12 can include picking up one or more chargingcomponent when the one or more charging component is in a state held bymagazine 15.

The flow diagram of FIG. 12 illustrates general features of a chargingmethod in one embodiment. Flow diagrams of FIGS. 13-19 illustratespecific examples of processes for use in charging methods that can beperformed in the performance of the general method generally describedwith reference to the flow diagram of FIG. 12.

One or more computer system of system 100, e.g., one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions. Referring to the flow diagram of FIG. 13, onegeneral aspect can include a method 1000 for charging a drill hole 18 inmining wall 16.

As shown in FIG. 13 and mining environment 500 of FIG. 1, method 1000can include at block 1002 placing with carrier 14 robot 102 at alocation in front of mining wall 16 adjacent to a front surface ofmining wall 16. The method can include at block 1006, performing aprescan to prescan mining wall 16 with camera system 104 to obtain oneor more image representation of a mining wall 16 and planning a scanningpath S (FIG. 3) for the robot using the one or more imagerepresentation. An image representation obtained at block 1006 can beregarded as a prescan image representation. System 100 can determinescanning path S in a manner that collision of robot 102 with majorsurfaces e.g., mining wall 16, sidewalls 24, floor 22, and ceiling 23can be avoided. A method can include at block 1010 performing a scan asset forth herein and detecting a drill hole 18 represented in one ormore image representation. The one or more image representation in whicha drill hole 18 can be detected at block 1010 can be a scan imagerepresentation in one embodiment as set forth herein. A method caninclude at block 1014 registering detected drill holes 18 detected atblock 1010 on drilling map 116 (FIG. 2). System 100 can be operative sothat system 100 registers drill hole 18 on drilling map 116 (FIG. 2) ifsystem 100 detects in an image representation of a representation of adrill hole 18.

The registering at block 1014 can link detected drill holes to indexedidentifiers for drill holes, which can provide linking between detecteddrill holes, including image data for the detected drill holes, toinformation in drilling map 116 (FIG. 2) of the various drill holes suchas detonator packages 510 associated to each drill hole 18. It has beennoted that a drilling map 116 can include a coordinates for drill holes18 that can be designed coordinated designated by a blast engineer.Embodiments herein recognize that drill holes 18 may not actually bedrilled at the precise locations of the designed coordinates. Whensystem 100 detects drill holes 18 by processing image data system 100can update or supplement the designed coordinate hole location with thecoordinates of the detected actual drill holes 18 detected by processingof image data.

Embodiments of system 100 herein can include automated operating modesand/or teleoperation operating modes. In one example of an automatedoperating mode, system 100 can charge drill holes 18 of mining wall 16in an order determined by a sequence of hole identifiers of drilling map116 (FIG. 2). As noted, drilling map 116 (FIG. 2) can include holeidentifiers and that each hole identifier can have an associateddetonator package 510 which can be designed and designated by a drillingengineer. These drill hole identifiers have a sequence, e.g., asdetermined by their identifier or can be designated to have a sequence.A method can include at block 1018 selecting with system 100 a detonatorpackage 510 based on drilling map 116 (FIG. 2). System 100 at block 1018can be charging a certain drill hole 18 of a sequence of drill holes andselecting of detonator package 510 at block 1018 can be based on anidentifier of the current drill hole 18 of a sequence of drill holesbeing charged. As noted drilling map 116 (FIG. 2) can includeinformation indicating a detonator package 510 associated to each drillhole 18, wherein each drill hole 18 can be identified with anidentifier. At block 1018, system 100 can look up a detonator package510 associated to a current drill hole 18 being subject to charging indrilling map 116 (FIG. 2) to identify detonator package 510 associatedto a current drill hole.

As an alternative to the processing described with reference to block1018 (automated operating mode selection of a detonator package) system100 can operate in a teleoperation mode so that selection of a detonatorpackage 510 based on an operator input. In one embodiment, operatorcomputer system 170 can include a user interface allowing an operator toobserve data of system 100 and to enter control information forselection of a detonator package 510. Reviewing data of system 100 (e.g.by visual observation) an operator can enter one or more operator inputusing a user interface of operator computer system 170 to designate aselection if a detonator package 510 for a current drill hole 18. It isnoted that drilling map 116 can include characterizing information thatassociates indexed drill holes 18 identified by identifiers to variousparticular detonator packages 510. In one embodiment, a selecting atblock 1018 can include selecting an indexed drill hole 18 of a miningwall 16 with a detonator package 510 associated to the drill hole 18based on an operator input. A method therefore allows an operator toselect an order of drill holes 18 to charge. The data on which aselection can be based can include e.g., location or dimensions of adrill hole 18 and/or image data such as image data of a one or moreimage representation obtained at block 1206.

A method can include at block 1022 assembling with magazine 15 a primer512 (FIG. 8) to detonator 514 (FIG. 8) having a signal wire 516 (FIG. 8)attached thereto to complete assembly of a selected detonator package510. A method can include at block 1022 picking up with tool assembly102T a detonator package 510. The picking up detonator package 510 caninclude moving with robot 102 charging hose 202 into position inrelation to magazine 15 to pick up a designated detonator package 510.For picking up a selected detonator package 510, charging hose 202 (FIG.9) (which in some embodiments can include an adapter fittable into amajor body of charging hose 202) can be shaped to be friction fit aboutprimer 512 (FIG. 9) of detonator package 510. As shown in FIG. 9,detonator package 510 can be held in fixed position by magazine 15 sothat when charging hose 202 is moved in relation to magazine 15 charginghose 202 (with or without adapter thereto) can be friction fit about theprimer 512 to associate detonator package 510 to charging hose 202.

With reference again to FIG. 13 and FIG. 1, a method 1000 can include atblock 1026 performing visual servoing using camera system 104 toaccurately position charging hose 202 (FIG. 10) for entry into a drillhole 18, e.g. by positioning charging hose 202 and one or moreadditional charging component held therein at an opening 218 of drillhole 18. During performance of visual servoing, system 100 can controlrobot 102 responsively to image data currently obtained using camerasystem 104 for entry of one or more charging component (e.g. charginghose 202 and detonator package 510 held therein) into drill hole 18.Image data currently obtained can refer to image data contemporaneouslyobtained. An image representation obtained during performance of visualservoing can have one or more different characteristic than an imagerepresentation obtained during a prescan. A visual servoing imagerepresentation can have a higher resolution than a prescan imagerepresentation. In some embodiments, performance of visual servoingimage capture can switch from a 3D format to a 2D format to permitfaster obtaining of image representations.

A method 1000 can include at block 1030 moving with tool assembly 102Tcharging hose 202 within drill hole 18 so that distal end 228 ofcharging hose 202 is positioned at a distal end 318 of drill hole 18(e.g., as shown in FIGS. 11A and 11B). Moving at block 1030 can includeinserting with tool assembly 102T charging hose 202 into drill hole 18.Moving charging hose 202 at block 1030 so that charging hose 202 ispositioned at a distal end of drill hole 18 can include e.g. movingcharging hose 202 with tool assembly 102T within drill hole 18 toperform pushing forward and/or retracting back charging hose 202 and/orrotating charging hose 202 to overcome an obstruction in drill hole 18to facilitate advance of charging hose 202 toward distal end 318. Toperform inserting of charging hose 202 in drill hole 18, tool assembly102T can be configured to hold charging hose 202. Tool assembly 102T,can be configured to move charging hose 202 forwardly and oppositelybackwardly illustrated by double-headed arrow 206, along longitudinalaxis 204 of charging hose 202. Tool assembly 102T, can be configured torotate charging hose 202 in first and second opposite directions asillustrated by curved double-headed arrow 208, about longitudinal axis204 of charging hose 202. System 100 can have an automated mode ofoperation in which the robot 102 with tool assembly 102T automaticallyperforms a charging component positioning procedure for controlling aposition of one or more charging component e.g. charging hose 202.System 100 can have a teleoperation mode of operation in which the robotwith tool assembly 102T performs the charging component positioningprocedure for controlling the position of the one or more chargingcomponent based on one or more operator input. In one embodiment, thecomponent positioning procedure can be a procedure for moving one ormore charging component from an opening 218 of drill hole 18 to a distalend 318 (FIGS. 11A and 11B) of drill hole 18, inclusive of anyobstruction avoidance component positioning routine performed for movingone or more charging component from an opening 218 of drill hole 18 to adistal end 318 of drill hole 18.

As illustrated in reference to FIG. 11B a method 1000 can include atblock 1034 system 100 with use of a pump (not shown), pumping anexplosive material 226 through charging hose 202 and into drill hole 18and retracting with tool assembly 102T charging hose 202 until charginghose 202 is retracted completely from drill hole 18.

A method 1000 can include at block 1038 system 100 determining whetherall detected drill holes 18 are filled. On completion of each explosivematerial fill, drilling map 116 (FIG. 2) can be updated to include anindicator of the completed step. Accordingly, system 100 can determinewhether all drill holes 18 are filled by examining drilling map 116(FIG. 2). If system 100 determines at block 1038 that all drill holes 18have not been filled, system 100 can return to block 1018 to selectanother detonator package 510. In accordance with method 1000 of theflow diagram of FIG. 13, a next detonator package 510 and a next drillhole 18 to charge can be based on a sequence of drill holes 18 andassociated detonator packages 510 that have been recorded in drillingmap 116 (FIG. 2). If system 100 determines at block 1038 that all drillholes 18 have been filled, a method can proceed to block 1042. A methodcan include at block 1042 placing with carrier 14 robot 102 at a nextcharging location, e.g., at a next mining wall 16.

In the method 1000 of the flow diagram of FIG. 13 along with referenceto FIG. 1, carrier 14 can place the robotic charging system having robot102 in front of a mining wall 16, where drill holes 18 are drilled. Acamera system 104, for example, a 3D camera system on the robot 102 canquickly rotate to scan the environment surrounding robot 102, and system100 can use the image data obtained from camera system 104 to plan acollision free scanning path S (FIG. 3) for the robot 102. Camera system104 can scan mining wall 16 using the scanning path S (FIG. 3), andsystem 100 can collect point cloud data of each frame along the path toobtain a 3D image representation of a scene, then detect drill hole 18locations. System 100 can determine whether detected drill holes 18match to drill hole representations 18R of drilling map 116 (FIG. 2).Drilling map 116 (FIG. 2) can provide information of the rough relativelocation of the drill holes 18 and drilling map 116 (FIG. 2) can alsoprovide information identifying a detonator package 510 to each drillhole 18. System 100 can register each detected drill hole location todrilling map 116. For performing a registration, system 100 can raise aflag in drilling map 116 for each drill hole representation 18R ofdrilling map 116 (FIG. 2) to indicate that drill hole 18 correspondingto the drill hole representation 18R has been located. Image dataobtained using camera system 104 can be associated to each drill holerepresentation 18R of drilling map 116 (FIG. 2). After registering thedetected drill holes 18 to the drilling map 116 at block 1014 the system100 can automatically select the drill hole 18 and correspondingdetonator package 510 to perform charging. A magazine 15 can holddetonator packages 510. Magazine 15 can automatically assemble a primer512 to a detonator 514 (e.g. a signal tube) to complete assembly of adetonator package 510 (block 1022). A robot 102 can pick up theassembled detonator package 510 by way of inserting detonator package510 into the tip of charging hose 202. Based on the detected location ofa current drill hole 18, robot 102 can move charging hose 202 toapproach to drill hole 18 on mining wall 16. Then, robot 102 can performvisual servoing to accurately position charging hose 202 at the opening218 of drill hole 18 on mining wall 16. Once the positioning iscompleted, a tool assembly 102T of robot 102 can insert charging hose202 into drill hole 18 with the detonator package 510. System 100 candetect if there is an obstruction in drill hole 18. If there is anobstruction in drill hole 18, tool assembly 102T can move charging hose202 within drill hole 18 to attempt to overcome the obstruction, e.g.push forward and retract back, rotate to overcome the obstruction. Aftercharging hose 202 reaches the end of drill hole 18, the system 100 canpump explosive material into drill hole 18 and tool assembly 102T canretract charging hose 202 until drill hole 18 is filled up and charginghose 202 is retracted completely out of drill hole 18. Then system 100can work on the next detected drill hole 18. After all the detecteddrill holes 18 represented in drilling map 116 are filled, the chargingprocess is completed in a current location. Carrier 14 can place therobot 102 at a next location.

One or more computer system of system 100, e.g. one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions. Referring to the flow diagram of FIG. 14, onegeneral aspect can include a method 1100 for charging a drill hole 18 ina mining wall 16.

Referring to the flowchart of FIG. 14 and mining environment 500 of FIG.1, one embodiment of the present disclosure includes a method 1000 forcharging a plurality of drill holes 18 in mining wall 16.

The method can also include at block 1110 (and with reference to FIG.9), obtaining with tool assembly 102T of robot 102 detonator package 510in a hollow end of charging hose 202. The method can also include atblock 1114 (and with reference to FIG. 10), positioning with toolassembly 102T charging hose 202 having the detonator package 510 forentry of charging hose 202 having detonator package 510 into drill hole18 (e.g., for drill hole entry), e.g. at a position adjacent to anopening 218 of drill hole 18. The method can also include at block 1118(and with reference to FIGS. 11A and 11B) moving with tool assembly 102Tcharging hose 202 along a length of drill hole 18 so that detonatorpackage 510 and the end of charging hose 202 is disposed at a distal end318 (FIGS. 11A and 11B) of drill hole 18. The method can also include atblock 1122 (and with reference to FIGS. 11A and 11B) feeding explosivematerial 226 (FIG. 12) into charging hose 202 (in the direction of arrow222, 11A and 11B) to deposit detonator package 510 at the end of drillhole 18 and deposit explosive material 226 (11A and 11B) along thelength of drill hole 18 while removing charging hose 202 (in a backwarddirection as indicated by arrow 231, 11A and 11B) from drill hole 18.Other embodiments may include corresponding computer systems, apparatus,and computer programs recorded on one or more computer storage devices,each configured to perform the actions of the methods.

With reference again to FIG. 1, system 100 can perform the positioningwith tool assembly 102T of charging hose 202 at block 1114 based on aprocessing of one or more image representation obtained with camerasystem 104. In one embodiment, an obtaining of the one or more imagerepresentation at block 1114 (FIG. 14) can include performing one ormore of a prescan, a scan, or a visual servoing. During performance of aprescan, robot 102 with camera system 104 can obtain one or more prescanimage representation. For example, system 100 can process one or moreprescan image representation such as low resolution one or more prescanimage representation obtained during a prescan (e.g., by rotating toolassembly about a generally fixed location to image mining wall 16). Theprescan may be used to determine a scan path S (FIG. 3) which canfacilitate robot 102 avoiding collision with obstacles in miningenvironment 500 such as floor 22, ceiling 23, sidewalls 24 (FIG. 3), andmining wall 16. During performance of a scan, system 100 can controlrobot 102 with camera system 104 to move along scan path S to allowsystem 100 to obtain with camera system 104 one or more scan imagerepresentation of mining wall 16 such as one or more high resolutionscan image representation, which may be obtained while avoidingcollision with the mining environment.

In one embodiment, a prescan image representation can have lowerresolution than a subsequent scan image representation. In oneembodiment, the prescan image representation can be obtained with camerasystem 104 being positioned at a further distance from mining wall 16than the position of camera system 104 relative to mining wall 16 forobtaining the scan image representation. A subsequently obtained scanimage representation, in one embodiment, can include image data morefully representing an opening 218 of and/or an interior of drill hole 18than a prescan image representation. For performing a scan, robot 102can move camera system 104 to various positions proximate to an opening218 of drill hole 18 in a manner that detailed 3D point cloud image datarepresenting an opening 218 and/or an interior of drill hole 18 can beobtained, e.g., so that an orientation and/or a depth of drill hole 18may be determined.

During performance of visual servoing as described in greater detailherein, robot 102 with tool assembly 102T including camera system 104can position one or more charging component for insertion into drillhole 18 based on currently obtained visual servoing one or more imagerepresentation or based on prior obtained scan image representation. Thecurrent one or more image representation can be a contemporaneouslyobtained one or more image representation contemporaneously obtained bysystem 100. During performance of visual servoing, system 100 can switchfrom obtaining three dimensional image representations to obtaining twodimensional image representations for faster image processing.

In one embodiment, with reference to the flow diagram of FIG. 14obtaining at block 1110 can be performed after performing of a pre-scanand a scan and prior to performing of visual servoing (system 100 canperform visual servoing with charging hose 202 being held by toolassembly 102T in one embodiment).

With reference again to FIG. 1, implementations of the presentdisclosure may include one or more of the following features. The methodwhere the obtaining with tool assembly 102T attached to the robotic armdetonator package 510 in the hollow end of charging hose 202 from amagazine supporting a plurality of detonator packages 510 may includecoordinating tool assembly 102T to magazine 15 supporting plurality ofdetonator packages 510. The method may further include magazine 15operative for assembling a plurality of primers 512 onto a plurality ofdetonators 514 (FIG. 8). A signal wire 516 (FIG. 8) may be attached toeach primer 512 (FIG. 8). The method may include moving charging hose202 where tool assembly 102T is operative to rotate charging hose 202back and forth around longitudinal axis 204 of charging hose 202 toovercome an obstruction of feeding charging hose 202 in drill hole 18(such as may be imposed e.g., by obstruction 235 of drill hole 18 shownin FIGS. 21 and 22). The method may include using a camera system 104which can include a 3D camera system. The method where the processing isbased on a drilling map 116 which can have recorded therein location andother information regarding a plurality of drill holes 18. The methodwhere the processing is performed off site from the obtaining therepresentation of the mining wall having the plurality of drill holes18. The method where obtaining the one or more image representationincludes merging together image data from a plurality of imagerepresentations representing the plurality of portions of mining wall16. Implementations of the described techniques may include hardware, amethod or process, or computer software on a computer-accessible medium.

System 100 can have an automated mode of operation in which the robot102 with tool assembly 102T automatically performs a charging componentpositioning procedure for controlling a position of one or more chargingcomponent, e.g., charging hose 202. System 100 can have a teleoperationmode of operation in which the robot with tool assembly 102T performsthe charging component positioning procedure for controlling theposition of the one or more charging component based on one or moreoperator input.

With reference to the flow diagrams of FIGS. 15-17 there is set forthherein a system 100 comprising a robot 102 having a tool assembly 102T,wherein system 100 is configured to operate in an automated mode ofoperation in which robot 102 with tool assembly 102T automaticallyperforms a charging component positioning procedure for controlling aposition of one or more charging component; and wherein system 100 isconfigured to operate in a teleoperation mode of operation in which therobot with the tool assembly performs the charging component positioningprocedure for controlling the position of the one or more chargingcomponent based on one or more operator input. In one embodiment, system100 can be configured to transition from the automated mode of operationto the teleoperation mode of operation in response to a sensed conditionbeing sensed. In one embodiment, system 100 can be configured totransition from the automated mode of operation to the teleoperationmode of operation in accordance with a manual override feature based onone or more operator input.

One or more computer system of system 100, e.g., one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions. Referring to the flow diagram of FIG. 15, onegeneral aspect can include a method 1300 for positioning a chargingcomponent, e.g. one or more charging component.

Referring to the flow diagram of FIG. 15, e.g., implemented by one ormore system 100 in mining environment 500 of FIG. 1, method 1200 caninclude at block 1202 operating a system 100 in an automated mode ofoperation and at block 1206 operating a system 100 in a teleoperationmode of operation. At block 1202 operating a system 100 in an automatedmode of operation can be a mode of operation in which robot 102 with thetool assembly 102T automatically performs a charging componentpositioning procedure for controlling a position of one or more chargingcomponent. At block 1206 operating a system 100 in a teleoperation modeof operation can be mode of operation in which robot 102 in oneembodiment with the tool assembly 102T performs the charging componentpositioning procedure for controlling the positioning of the one or morecharging component based on one or more operator input. An automatedmode and/or a teleoperation mode can include actions other than chargingcomponent positioning. For example as explained in reference to FIG. 13selection of next drill hole 18 can be performed either in an automatedmode of operation of a teleoperation mode of operation.

In one embodiment the one or more charging component can be one or moreof a charging hose 202, detonator package 510, a primer 512, a detonator514, a signal wire 516, or explosive material 226. In one embodiment,for a time that the automated mode of operation is active the robot 102with the tool assembly 102T performs the charging component positioningprocedure independent of any current operator input.

In one embodiment, the charging component positioning procedure is apositioning procedure for positioning one or more charging component forentry into a drill hole 18. In one embodiment, the charging componentpositioning procedure is a positioning procedure for moving one or morecharging component from an opening 218 of a drill hole 18 through to adistal end of the drill hole 18 inclusive of any obstruction avoidancecharging component positioning routine performed for moving the one ormore charging component from an opening 218 of a drill hole 18 throughto a distal end of the drill hole 18. In one embodiment, the chargingcomponent positioning procedure is a positioning procedure for avoidingan obstruction within a drill hole 18. In one embodiment, the toolassembly 102T includes a camera system 104, and system 100 deactivatesthe automated mode of operation based on one or more imagerepresentation obtained with the camera system 104.

In one embodiment, system 100 is configured for operation in anautomated operating mode to perform a charging component positioningprocedure to position one or more charging component for entry into adrill hole 18, and system 100 is configured to transition from theautomated mode of operation to the teleoperation mode of operation inresponse to a sensed condition being sensed, the sensed condition beingthe condition that the one or more charging component is not positionedproperly for drill hole entry into a drill hole 18, the sensed conditionbeing determined by processing of one or more image representationobtained with the camera system 104.

In one embodiment, the system 100 is configured for operation in anautomated operating mode to perform a charging component positioningprocedure for moving one or more charging component from an opening 218of a drill hole 18 to a distal end 318 of a drill hole 18 inclusive ofany obstruction avoidance charging component positioning routine andsystem 100 is configured to transition from the automated mode ofoperation to the teleoperation mode of operation in response to a sensedcondition being sensed, the sensed condition being the condition thatthe one or more charging component has not reached a distal end 318 ofthe drill hole 18.

In one embodiment, the system 100 in the teleoperation mode activates inresponse to one or more user input an obstruction avoidance chargingcomponent positioning procedure selected from a group comprising (a)pushing in (b) retracting (c) rotating and (d) wiggling.

In one embodiment, the system 100 with tool assembly 102T in theteleoperation mode provides motion to a charging component thatcorresponds to motion that is imparted by an operator to an actuator ofan operator computer system. In one embodiment, the one or more operatorinput is an input of a supervisor operator entered into a user interfaceof a remote operator computer system located remoted from the robot 102.

In one embodiment, system 100 can have a manual override feature and canbe configured so that system 100 transitions from an automated mode to ateleoperation mode based on an input of an operator entered using a userinterface of operator computer system 170. Thus, system 100 operating inan automated mode of operation to perform a component positioningprocedure for positioning one or more charging component for entry intoa drill hole 18 can transition to a teleoperation mode of operation forpositioning one or more charging component for entry into a drill hole18 based on an input of an operator entered using a user interface ofoperator computer system 170 to activate a manual override of theautomated operating mode. System 100 operating in an automated mode ofoperation for moving one or more charging component from an opening 218of a drill hole 18 to a distal end 318 of drill hole 18 (inclusive ofany obstruction avoidance component positioning routine) can transitionto a teleoperation mode of operation for moving one or more chargingcomponent from an opening 218 of a drill hole 18 to a distal end 318 ofdrill hole based on an input of an operator entered using a userinterface of operator computer system 170 to activate a manual overrideof the automated operating mode.

As has been noted operator computer system 170, in one embodiment, caninclude local operator computer system and supervisor operator computersystem and be distributed in both local and remote locations to permitoperation by both one or more operator at a local location and one ormore operator at a remote location. In one embodiment, system 100 can beconfigured so that the remote supervisor operator computer system haspriority over the local operator computer system. In one embodiment,system 100 can be configured so that system 100 disables a localoperator computer system based on one or more operator input enteredinto a user interface of a remote supervisor operator computer system ofoperator computer system 170.

The flow diagram of FIG. 15 illustrates general features of a chargingmethod in one embodiment wherein a system 100 can transition between anautomated mode of operation and a teleoperation mode of operation. Flowdiagrams of FIGS. 16 and 17 illustrate specific examples of chargingmethods that can be performed in the performance of the general methodgenerally described with reference to the flow diagram of FIG. 12wherein a system 100 can transition between an automated mode ofoperation and a teleoperation mode of operation. Specific embodiments ofa system 100 operating in an automated mode of operation and ateleoperation mode of operation are set forth in reference to FIGS.16-17.

In one embodiment, as illustrated with reference to FIG. 16 thecomponent positioning procedure can include a procedure for positioningone or more charging component for entry into a drill hole 18 (FIG. 10).

One or more computer system of system 100, e.g., one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions. Referring to the flow diagram of FIG. 16, onegeneral aspect can include a method 1300 for positioning one or morecharging component.

Referring to the flow diagram of FIG. 16, e.g., implemented by one ormore system 100 in mining environment 500 of FIG. 1, method 1300 caninclude at block 1302 performing visual servoing with camera system 104to position one or more charging component held by tool assembly 102Tfor drill hole entry e.g. at a position adjacent to an entrance of adrill hole 18. In performing visual servoing, system 100 can control aposition of robot 102 for insertion of one or more charging componentinto drill hole 18 based on current image data of the system, e.g.,image data of one or more image representation contemporaneouslyobtained using camera system 104. A charging component can be e.g. oneor more of charging hose 202 a detonator package 510 a component of adetonator package 510 or explosive material 226. A method can include atblock 1306, system 100 determining whether a positioning of the one ormore charging component resulting from performing visual servoing issuccessful. For example, system 100 can determine if the obtained imagedata obtained during visual servoing satisfies predetermined criteriafor the image data representing a drill hole 18. Image data representingother than a drill hole 18 can indicate that a robot is misaligned withrespect to a drill hole 18 indicating a possible error in positioning ofone or more charging component for entry into drill hole 18. In oneembodiment functions described with reference to block 1302 can beperformed with system 100 operating in an automated mode of operation asset forth herein.

A method can include at block 1310, on determination at block 1306 thata positioning is not successful, controlling robot 102 to position theone or more charging component held by robot 102 based on one or moreinput obtained from an operator. Thus, in one embodiment functionsdescribed with reference to block 1310 as set forth in the flow diagramof FIG. 16 can be performed with system 100 operating in a teleoperationmode of operation as set forth herein. At block 1310 in one embodimentsystem 100 can transition from an automated mode of operation to ateleoperation mode of operation. An operator can teleoperate the robot102 to position the one or more charging component, e.g., one or more ofcharging hose 202, detonator package 510, one or more component ofdetonator package 510 or explosive material 226 for entry into drillhole 18, e.g., by positioning the one or more charging component at theentrance of a drill hole 18. Referring to FIG. 20 operator computersystem 170 can include a user interface. A physical form view of anoperator computer system 170 can include a display 1702 in combinationwith an actuator provided by a haptic motion controller 1706 (FIG. 20).A Haptic motion controller can provide tactile feedback to an operator.Haptic motion controller 1706 (FIG. 20) can be a PHANTOM OMNI® motioncontroller available from Sensable Technologies. In one embodiment,system 100 can be operative so that tool assembly 102T provides motionto one or more charging component that corresponds to motion that isimparted to motion imparted by an operator to haptic motion controller1706 (FIG. 20). For example, in one embodiment, imparting a force tomember 1708 of haptic motion controller 1706 along axis 704 of member1708 in a direction of arrows 706 can result in movement of one or morecharging component in a corresponding direction indicated by arrows 206(FIG. 1, FIGS. 6-7) along axis 204 (FIG. 1, FIGS. 6-7). Imparting aforce to member 1708 of haptic motion controller 1706 rotationally aboutaxis 704 of member 1708 in a direction of arrows 708 can result inmovement of one or more charging component in a corresponding rotationaldirection indicated by arrows 208 (FIG. 1, FIGS. 6-7) about axis 204(FIG. 1, FIGS. 6-7). Operator computer system 170 having a userinterface can be located in an underground area 12 and/or external tounderground area 12. An operator can input robot position control datausing haptic motion controller 1706 on observing current image dataobtained from camera system 104 which current image data can bedisplayed on display 1702. System 100 can control a position of robot102 holding one or more charging component based on one or more controlinput obtained from an operator and entered by an operator into a userinterface of operator computer system 170.

With reference again to FIG. 16, a method can include at block 1314system 100 determining that positioning is complete by confirming thatthe one or more charging component is positioned for entry into a drillhole 18. Such determining can include e.g. system 100 performing patternrecognition processing using one or more image representation obtainedwith use of camera system 104 to confirm that a drill hole 10 is alignedto one or more charging component. On the confirmation the one or morecharging component has been positioned for drill hole entry system 100can be operative to proceed to move one or more charging componentwithin a drill hole 18 in accordance with, for example, a chargingcomponent positioning procedure for positioning one or more chargingcomponent within a drill hole 18 e.g. from an opening 218 of a drillhole 18 to a distal end 318 of a drill hole 18.

As noted above, the robot charging system can be teleoperated (locallyor remotely) if visual servoing guidance which can be performed in anautomated operating mode cannot successfully position a chargingcomponent e.g., charging hose 202 relative to drill hole 18. Such atechnique may improve the robustness and reliably of system 100, forexample, even if there is error in the automatic drill hole detectionand positioning, system 100 may still complete the positioning of acharging component with operator's guidance.

System 100 can have an automated mode of operation in which the robot102 with tool assembly 102T automatically performs a charging componentpositioning procedure for controlling a positioning of one or morecharging component e.g. charging hose 202. System 100 can have ateleoperation mode of operation in which the robot 102 with toolassembly 102T performs the charging component positioning procedure forcontrolling the positioning of the one or more charging component basedon one or more operator input. In one embodiment, as set forth withreference to the flow diagram of FIG. 16, the component positioningprocedure can be a procedure for positioning one or more chargingcomponent, e.g. charging hose 202, a detonator package 510, one or morecomponent of detonator package 510, or explosive material 226 for drillhole entry (FIG. 10).

In the embodiment described with reference to the flow diagram of FIG.16, system 100 is configured to operate in each of an automated mode ofoperation and a teleoperation operating mode of operation and cantransition from an automated mode of operation to a teleoperation modeof operation based on a sensed condition that can be sensed at block1306. The sensed condition can be e.g. the determination that apositioning has been unsuccessful and can be based on a result of aprocessing of one or more image representation obtained with camerasystem 104 during performance of visual servoing. In one embodiment,system 100 can have a manual override feature and can be configured sothat system 100 transitions from an automated mode to a teleoperationmode based on an input of an operator entered using a user interface ofoperator computer system 170.

One or more computer system of system 100, e.g., one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions. Referring to the flow diagram of FIG. 17, onegeneral aspect can include a method 1400 for moving one or more chargingcomponent within a drill hole 18.

In reference to method 1400 illustrated in reference to the flow diagramof FIG. 17 along with FIG. 1, method 1400 can include at block 1402system 100 performing a charging component positioning procedure formoving one or more charging component from an opening 218 of drill hole18 to a distal end 318 of a drill hole 18. For performance of a chargingcomponent positioning procedure as set forth in reference to block 1402a tool assembly 102T can be operated to move one or more chargingcomponent (e.g. one or more of a charging hose, a detonator package 510,one or more component of a detonator package 510, explosive material226) within a drill hole 18 for performance of component positioningprocedure, e.g., to one or more of push forward charging hose 202 heldby tool assembly 102T, retract charging hose 202 or rotate charging hose202 to overcome an obstruction 235 (FIGS. 11A and 11B) during movementof a charging hose 202 within drill hole 18 from opening 218 to distalend 318 of drill hole 18. Referring to FIGS. 11A and 11B the one or morecomponent can include in one embodiment a charging hose 202 held by toolassembly 102T and a detonator package 510 held by charging hose 202.System 100 can be operating in an automated mode of operation duringperformance of block 1402 in a manner that selection of one or moreobstruction avoidance routine is automatic, e.g. one or more differentroutines can be automatically performed in e.g. a random sequence, apredetermined sequence or a sequence based on one or more sensedcondition, a sensed motion resistance or torque sensed by tool assembly102T using one or more sensor of sensors/sensing devices 108 duringperformance of an obstruction avoidance routine. A method can include atblock 1406 system 100 determining whether a one or more chargingcomponent has reached a distal end 318 of a drill hole 18. If yes system100 can proceed to block 1430 and drill hole movement for a currentdrill hole 18 can be determined to be complete. If no, system 100 canproceed to block 1410. For determining at block 1406 whether movement ofone or more charging component within a drill hole 18 is complete in oneembodiment system 100 can compare a detected position of charging hose202 based on a length of charging hose 202 offset from robot 102 to arecorded depth of a drill hole 18 recorded in drilling map 116. Arecorded depth of a drill hole 18 indicative of distal end 318 in oneembodiment can be a design parameter designed in by a blast engineer.For determining whether movement of one or more charging componentwithin a drill hole 18 is complete in one embodiment system 100 cancompare a detected position of one or more charging component based on alength of charging hose 202 offset from robot 102 to one or more imagerepresentation of a drill hole 18 that can include 3D point cloud datathat indicated a depth of drill hole 18. System 100 can be operative todetermine a distance of charging hose 202 offset from robot 102 when itis inserted based on data available from one or more moving mechanism oftool assembly 102T that indicates a length of charging hose 202currently extending from an end of tool assembly 102T.

A method 1400 can include at block 1410 system 100 performing anobstruction avoidance charging component positioning procedure based onone or more input obtained from an operator. Operator computer system170 can include a user interface having a display 1702 displaying a userinterface display screen as shown in FIG. 22. Using a user interface ofoperator computer system 170 an operator can select, e.g., by pointingand clicking with pointer 1718 an obstruction avoidance chargingcomponent positioning procedure such as (a) push (b) retract (c) rotate(d) wiggle (rotate back and forth) to handle one or more chargingcomponent e.g. including charging hose 202 to overcome the obstruction.A one or more obstruction avoidance component positioning procedure thatcan be performed at block 1410 can be similar to a one or moreobstruction avoidance routine that can be performed at block 1402 exceptthat whereas obstruction avoidance routines performed at block 1402 canbe automatically selected for activation by system 100, obstructionavoidance component positioning procedures performed at block 1410 canbe selected based on one or user input of an operator.

A method 1400 as set forth in the flow diagram of FIG. 17 can include atblock 1414 system 100 determining again whether one or more chargingcomponent e.g. including charging hose 202 has reached an end of a drillhole 18. If yes system 100 can proceed to block 1430 and chargingcomponent moving for a current drill hole can be determined to becomplete. If no at block 1410 an operator can teleoperate robot 102 tohandle one or more charging component e.g. including charging hose 202to overcome the obstruction. A method can include at block 1410 system100 performing with robot 102 charging component positioning based onspecific actuator control inputs of an operator so that motion of one ormore charging component corresponds to motion imparted to an actuator byan operator. Operator computer system 170 can include a user interfacehaving a display 1702 a physical form view of which is shown in FIG. 20.Using a user interface of operator computer system 170 an operator canmove one or more charging component e.g. including charging hose 202 ina manner specified by the operator using e.g., haptic motion controller1706 shown in FIG. 20. System 100 can be operative to control motion ofone or more charging component, e.g. one or more of charging hose 202,primer 512, detonator 514 or signal wire 516 so that motion of the oneor more charging component corresponds to motion imparted to a userinterface actuator of an operator computer system 170. For example,system 100 can be operative so that if an operator moves member 1708 ofhaptic motion controller 1706 axially in a direction of arrows 706 alongaxis 704 of member 1708, system 100 moves the charging component alongits axis, e.g. longitudinal axis 204 (FIG. 1, FIG. 6, FIG. 7). In oneparticular embodiment, system 100 can be operative so that one or morecharging component e.g. including charging hose 202 is moved by toolassembly 102T in or out axially along longitudinal axis 204 (FIG. 1) adistance proportional to a distance that operator moves member 1708 ofhaptic motion controller 1706 in or out in a direction indicated byarrows 706. System 100 can be operative so that if an operator movesmember 1708 of haptic motion controller 1706 rotationally about its axis704 in a direction of arrows 708, system 100 moves the one or morecharging component rotationally about its axis, e.g. longitudinal axis204 (FIG. 1, FIG. 6, FIG. 7). In one particular embodiment, system 100can be adapted so that one or more charging component e.g. includingcharging hose 202 is rotated by tool assembly 102T back or forth aboutlongitudinal axis 204 in a direction of arrows 208 (FIG. 1, FIG. 6, FIG.7) an amount proportional to an amount that operator rotates member 1708of haptic motion controller 1706 back or forth rotationally in adirection indicated by arrows 708 (FIG. 20). At block 1422 system 100can again determine whether a one or more charging component e.g.including charging hose 202 has reached a distal end 318 of drill hole18. If yes system 100 can proceed to block 1430 and charging componentmovement for a current drill hole 18 can be determined to be complete.If not system 100 can log in e.g., to drilling map 116 that a distal end318 of drill hole 18 has not been reached and a warning can be raised atblock 1526.

System 100 as illustrated in the flow diagram of FIG. 17 can have anautomated mode of operation in which the robot 102 with tool assembly102T automatically performs a charging component positioning procedurefor controlling a positioning of one or more charging component e.g.charging hose 202, detonator package 510, one or more component ofdetonator package 510 or explosive material 226. System 100 asillustrated in the flow diagram of FIG. 17 can have a teleoperation modeof operation in which the robot 102 with tool assembly 102T performs thecharging component positioning procedure for controlling the position ofthe one or more charging component based on one or more current operatorinput. In one embodiment, as set forth with reference to the flowdiagram of FIG. 17, blocks 1402, 1406, 1410, 1414, and 1418, thecomponent positioning procedure can be a procedure for moving one ormore charging component from an opening 218 of a drill hole 18 to adistal end 318 of drill hole 18.

In the embodiment described with reference to the flow diagram of FIG.17, blocks 1402, 1406, 1410, 1414, and 1418 system 100 is configured tooperate in each of an automated mode of operation (block 1402 and block1410) and a teleoperation operating mode of operation (block 1410 andblock 1418) and can transition from an automated mode of operation to ateleoperation mode of operation based on a sensed condition that can besensed at block 1406. The sensed condition can be, e.g., thedetermination at block 1406 that one or more charging component has notreached a distal end 318 of drill hole 18. In one embodiment, system 100can have a manual override feature and can be configured so that system100 transitions from an automated mode to a teleoperation mode based onone or more input of an operator entered using a user interface ofoperator computer system 170.

As set forth herein system 100 can have an automated mode of operationin which the robot 102 with tool assembly 102T automatically performs acharging component positioning procedure for controlling a position ofone or more of charging component e.g. charging hose 202, detonatorpackage 510, one or more component of detonator package 510, orexplosive material 226. System 100 can have a teleoperation mode ofoperation in which the robot 102 with tool assembly 102T performs thecharging component positioning procedure for controlling the position ofthe one or more charging component based on one or more operator input.In one embodiment, as set forth with reference to the flow diagram ofFIG. 17, blocks 1410, 1414, and 1418, the charging component positioningprocedure can be an obstruction avoidance charging component positioningprocedure for moving one or more charging component to avoid anobstruction within a drill hole 18.

In the embodiment described with reference to the flow diagram of FIG.17, blocks 1410, 1414, and 1418, system 100 is configured to operate ineach of an automated mode of operation (block 1410) and a teleoperationoperating mode of operation (block 410 and block 1418) and cantransition from an automated mode of operation to a teleoperation modeof operation based on a sensed condition that can be sensed at block1414 (an obstruction avoidance component positioning procedure can beselected during operation of system 100 in a teleoperation mode ofoperation at block 1410 and when the obstruction avoidance componentpositioning procedure is active system 100 can be regarded to beoperating in an automated operating mode of operation). The sensedcondition at block 1414 can be e.g., the determination that one or morecharging component has not reached a distal end 318 of a drill hole 18.The sensed condition at block 1414 can in addition or alternatively bee.g. a sensed motion resistance or torque sensed by tool assembly 102Tusing one or more sensor of sensors/sensing devices 108 duringperformance of an obstruction avoidance component positioning procedure.In one embodiment, system 100 can have a manual override feature and canbe configured so that system 100 transitions from an automated mode to ateleoperation mode based on one or more input of an operator enteredusing a user interface of operator computer system 170.

Referring overall to the flow diagram of FIG. 17, if a distal end 318 ofdrill hole 18 cannot be reached (determined at block 1406 e.g. based thedepth information of drill hole 18 which can be recorded in drilling map116 as set forth herein) with system 100 operating in in an automatedmode of operation (block 1402) system 100 can transition to ateleoperation mode of operation (1410) to permit selection of anobstruction avoidance charging component positioning procedure. With theselected charging component positioning procedure active (block 1410)system 100 can be regarded to be operating in an automated mode ofoperation. If a drill hole distal end 318 still cannot be reached (block1414) or an alternate condition sensed (e.g. motion resistance, torqueas set forth herein), system 100 can transition again to a teleoperationmode of operation and proceed to block 1418. At block system 100 canperform actuator corresponding motion control so that motion of one ormore charging component corresponds to motion imparted to a userinterface actuator as set forth herein. If a drill hole distal end 318still cannot be reached (block 1422) the system 100 can log in themissed drill hole 18 into drilling map 116 and raise a warning.

The method of FIG. 17 illustrates the flexibility and robustness of thesystem 100 for robot charging with teleoperation and robot programming.It also provides the quality control and monitoring the chargingprocess. The information obtained by system 100 can be available to anoperator (e.g. local operator and/or supervisor operator) in real timeon an ongoing basis as the information is obtained.

It will be recognized in connection with the flow diagrams of FIGS.15-17 that if system 100 transitions from an automated mode of operationto a teleoperation mode of operation during performance of a chargingcomponent positioning procedure, the procedure being performed (e.g.positioning one or more charging component for entry into a drill hole18, moving one or more charging component from an opening 218 to adistal end 318, positioning one or more charging component for avoidingan obstruction within a drill hole 18) at the time of the transitionneed not necessarily be completed by system 100 and if it is completedmay be completed with the system 100 operating in a teleoperation modeof operation and not the automated mode of operation.

One or more computer system of system 100, e.g., one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions. Referring to the flow diagram of FIG. 18, onegeneral aspect can include a method 1500 for registering a detecteddrill hole 18 detected within 3D point cloud image data with drillingmap 116.

Referring to the flow diagram of FIG. 18, e.g., implemented by one ormore system 100 in mining environment 500 of FIG. 1, method 1500 caninclude at block 1502 system 100 obtaining with camera system 104 animage representation of a mining wall 16 and detecting drill holes 18 inthe one or more image representation. In one embodiment, the one or moreimage representation can be provided by one or more 3D point cloud imagedata representation. A method can include at block 1504 system 100registering the detected drill holes to drilling map 116 as set forthherein. A method 1500 can include at block 1506 system 100 comparingdetected drill hole representations of one or more image representationof drilling map 116. A method 1500 can include at block 1510 system 100determining whether all drill holes 18 represented in drilling map 116have been detected within the one or more image representation obtainedat block 1502. If yes, system 100 can proceed to block 1518 where system100 can determine that drill hole registration is complete. If not, atblock 1514 an operator can select missed drill holes from one or moreimage representation of a mining wall 16 (e.g., using the featuredescribed with reference to FIG. 21) and register the operator selecteddrill holes 18 to drilling map 116, e.g., registering missed drill holes18 from one or more image representation based on an operator input.

Referring to FIG. 21 and system 100 of FIG. 1, operator computer system170 can include a user interface with display 1702. System 100 candisplay on display image data of the one or more image representationcorresponding to a missed drill hole representation of drilling map 116.System 100 can be configured to allow an operator e.g., by movement ofpointer 1718 to manipulate one or more highlight 1710 of display 1702 toindicate a region of interest of displayed image data to indicate imagedata that should have been detected by system 100 as image datarepresenting a drill hole 18. System 100 can be configured so that adrill hole representation of one or more image representation may belower quality and may not be detected as a drill hole representationautomatically by system 100 (e.g. because predetermined criteria forimage data representing a drill hole 18 are not satisfied).Notwithstanding, where system 100 does not detect a drill holerepresentation as a drill hole system 100 may flag such undetected drillhole representation as a possible drill hole representation that can bedetected as a drill hole representation on the entry of an operatorhighlight 1710. Referring to FIG. 21 displayed image data displayed onan operator computer system 170 representing a possible location ofdrill hole 18 can be superimposed on displayed image data representing amining wall 16, e.g., by display of highlight 1710, and a user can clickon the highlight to designate the representation as a detected drillhole 18. System 100 can be adapted so that an operator can point andclick a highlight 1710 using a pointer 1718 to select the location of adrill hole 18 represented in displayed image data as displayed ondisplay 1702. In the example of FIG. 21 an operator can control pointer1718 to move highlight 1710 to a selected one of the possible drill holerepresentations 18D shown and then perform a click actuation to selectthe drill hole 18 represented by the representation 18D that ishighlighted by highlight 1710 to permit system 100 to designate theselected drill hole 18 as a detected drill hole 18. Responsivelythereto, system 100 can designate the selected drill hole 18 as adetected drill hole 18. The selected drill hole 18 selected based on oneor more operator input can be registered to drilling map 116.

One or more computer system of system 100, e.g., one or more computersystem of robot 102, robot controller 150, operator computer system 170or backend computer system 160 can be configured to perform particularoperations or actions by virtue of having software, firmware, hardware,or a combination of them installed on the one or more computer that inoperation causes or cause the one or more computer to perform theactions. One or more computer program can be configured to performparticular operations or actions by virtue of including instructionsthat, when executed by data processing apparatus, cause the apparatus toperform the actions. Referring to the flow diagram of FIG. 19, onegeneral aspect can include a method 1600 for using a robot in a miningenvironment.

Referring to the flow diagram of FIG. 19 along with FIG. 1, method 1600can include at block 1602 obtaining with camera system 104 one or moreimage representation of mining wall 16 in a state prior to blasting ofthe mining wall 16 using explosive material. The one or more imagerepresentation can include one or more 3D point cloud imagerepresentation. A method can include at block 1606 obtaining with camerasystem 104 one or more image representation of mining wall 16 in a statesubsequent to blasting. The one or more image representation can includeone or more 3D point cloud image representation. The obtaining of one ormore image representation at block 1602 and/or block 1606 can includeone or more prescan image representation as set forth herein. Carrier 14of system 100 can place robot 102 at appropriate locations in relationto mining wall 16.

A method can include at block 1622 system 100 comparing one or moreimage representation obtained at block 1602 (prior to blasting) to oneor more image representation obtained at block 1606 (subsequent toblasting). A method can include at block 1610 system 100 outputtinginformation of a result of the comparing at block 1610. The informationsubject to outputting can be e.g., information indicating a result ofthe blasting. The information output can be information indicatinggeometric changes resulting from the blasting. One or more imagerepresentation obtained at block 1602 and block 1610 can include threedimensional information characterizing mining wall 16 (including in oneembodiment associated ceiling 23 floor 22 and sidewalls 24) before andafter blasting. Output information output at block 1614 can include dataindicating a volume of mining wall material removed from mining wall 16as a result of the blasting.

With reference to FIGS. 1 and 3, system 100 can monitor production bymeasuring the front mining wall 16 and/or associated sidewall 24 ceiling23 or floor 22 geometric changes. After carrier 14 places robot 102 infront of mining wall 16 and once system 100 completes the scanning ofthe surrounding environment before and after blasting, system 100 cancompare one or more image representation (e.g. which can be one or more3D point cloud image representation of mining wall 16 which can includein one embodiment associated representations of a ceiling 23, sidewalls24 and floor 22 as shown in FIG. 3) before the last blasting and afterthe last blasting to calculate the changes of mining wall 16. The miningproduction information obtained by system 100 can be available tooperator and supervisor in real time (e.g. delay limited to dataprocessing delay) on an ongoing basis as the information is obtained.

In one embodiment, a method can include using more than one robot in amining environment. FIG. 23 illustrates more than one robot working on amining wall 16. Referring to FIG. 23 a mining environment 500 caninclude a first robot 102 at a first location “A” and a second robot 102at a second location “B”. System 100 can be operative so that each ofthe robot at location “A” and location “B” performs work simultaneously.

In one embodiment, system 100 can be operative so that each of firstrobot 102 at a first location “A” and second robot 102 at a secondlocation “B” perform a common task simultaneously. In one embodiment,system 100 can be operative so that each of first robot 102 at firstlocation “A” and second robot 102 at second location “B” performs aprescan simultaneously to obtain one or more prescan imagerepresentation of mining wall 16 simultaneously. In one embodiment,system 100 can be operative so that each of first robot 102 at firstlocation “A” and second robot 102 at second location “B” performs a scansimultaneously to obtain one or more scan image representation of miningwall 16. In one embodiment, system 100 can be operative so that each offirst robot 102 at first location “A” and second robot 102 at secondlocation “B” performs visual servoing for positioning one or morecharging component for insertion into drill hole 18 to obtain one ormore visual servoing image representation. Where operative so that thefirst and second robots with respect to mining wall 16 simultaneously,system 100 can be operative so that the first and second robots arerestricted from working within a threshold distance away from another soas to prevent collision between the first and second robots.

In one embodiment, system 100 can be operative so that first robot 102at first location “A” and second robot 102 at second location “B”perform a different respective tasks simultaneously. In one embodiment,system 100 can be operative so that first robot 102 at first location“A” performs a prescan simultaneously to obtain one or more prescanimage representation of mining wall 16 simultaneously while second robot102 at second location “B” performs a scan to obtain one or more scanimage representation of mining wall 16. In one embodiment, system 100can be operative so that first robot 102 at first location “A” performsa prescan simultaneously to obtain one or more prescan imagerepresentation of mining wall 16 simultaneously while and second robot102 at second location “B” performs visual servoing for positioning oneor more charging component for entry into a drill hole 18 to obtain oneor more visual servoing image representation of mining wall 16. In oneembodiment, system 100 can be operative so that first robot 102 at firstlocation “A” performs a scan to obtain one or more scan imagerepresentation of mining wall 16 simultaneously while second robot 102at second location “B” performs visual servoing for positioning one ormore charging component for entry into a drill hole 18 to obtain one ormore visual servoing image representation of mining wall 16. Whereoperative so that first and second robots with respect to a mining wall16 simultaneously, system 100 can be operative so that the first andsecond robots are restricted from working within a threshold distanceaway from another so as to prevent collision between the first andsecond robots.

Referring again FIG. 12, all of the functions described with referenceto the flow diagram of FIG. 12 in one embodiment can be performed withsystem 100 operating in an automated mode of operation as set forthherein (wherein functions performed by system 100 can be independent ofany current operator input with the automated operating mode ofoperation active) without transitioning to a teleoperation mode ofoperation. Referring still to FIG. 12, all of the functions describedwith reference to the flow diagram of FIG. 12 in one embodiment can beperformed with system 100 operating in a teleoperation operating mode ofoperation without transitioning to an automated mode of operation.Referring still to FIG. 12, the functions described with reference tothe flow diagram of FIG. 12 in one embodiment can be performed withsystem 100 transitioning one or more times between an automated mode ofoperation and a teleoperation mode of operation.

Referring again FIG. 13, all of the functions described with referenceto the flow diagram of FIG. 13 in one embodiment can be performed withsystem 100 operating in an automated mode of operation as set forthherein (wherein functions performed by system 100 can be independent ofany current operator input with the automated operating mode ofoperation active) without transitioning to a teleoperation mode ofoperation. Referring still to FIG. 13, all of the functions describedwith reference to the flow diagram of FIG. 13 in one embodiment can beperformed with system 100 operating in a teleoperation operating mode ofoperation without transitioning to an automated mode of operation.Referring still to FIG. 13, the functions described with reference tothe flow diagram of FIG. 13 in one embodiment can be performed withsystem 100 transitioning one or more times between an automated mode ofoperation and a teleoperation mode of operation.

Referring again FIG. 14, all of the functions described with referenceto the flow diagram of FIG. 14 in one embodiment can be performed withsystem 100 operating in an automated mode of operation as set forthherein (wherein functions performed by system 100 can be independent ofany current operator input with the automated operating mode ofoperation active) without transitioning to a teleoperation mode ofoperation. Referring still to FIG. 14, all of the functions describedwith reference to the flow diagram of FIG. 14 in one embodiment can beperformed with system 100 operating in a teleoperation operating mode ofoperation without transitioning to an automated mode of operation.Referring still to FIG. 14, the functions described with reference tothe flow diagram of FIG. 14 in one embodiment can be performed withsystem 100 transitioning one or more times between an automated mode ofoperation and a teleoperation mode of operation.

Referring again FIG. 18, all of the functions described with referenceto the flow diagram of FIG. 18 in one embodiment can be performed withsystem 100 operating in an automated mode of operation as set forthherein (wherein functions performed by system 100 can be independent ofany current operator input with the automated operating mode ofoperation active) without transitioning to a teleoperation mode ofoperation. Referring still to FIG. 18, all of the functions describedwith reference to the flow diagram of FIG. 18 in one embodiment can beperformed with system 100 operating in a teleoperation operating mode ofoperation without transitioning to an automated mode of operation.Referring still to FIG. 18, the functions described with reference tothe flow diagram of FIG. 18 in one embodiment can be performed withsystem 100 transitioning one or more times between an automated mode ofoperation and a teleoperation mode of operation.

Referring again FIG. 19, all of the functions described with referenceto the flow diagram of FIG. 19 in one embodiment can be performed withsystem 100 operating in an automated mode of operation as set forthherein (wherein functions performed by system 100 can be independent ofany current operator input with the automated operating mode ofoperation active) without transitioning to a teleoperation mode ofoperation. Referring still to FIG. 19, all of the functions describedwith reference to the flow diagram of FIG. 19 in one embodiment can beperformed with system 100 operating in a teleoperation operating mode ofoperation without transitioning to an automated mode of operation.Referring still to FIG. 19, the functions described with reference tothe flow diagram of FIG. 19 in one embodiment can be performed withsystem 100 transitioning one or more times between an automated mode ofoperation and a teleoperation mode of operation.

Referring again FIG. 23, all of the functions described with referenceto FIG. 23 in one embodiment can be performed with system 100 operatingin an automated mode of operation as set forth herein (wherein functionsperformed by system 100 can be independent of any current operator inputwith the automated operating mode of operation active) withouttransitioning to a teleoperation mode of operation. Referring still toFIG. 23, all of the functions described with reference to the flowdiagram of FIG. 23 in one embodiment can be performed with system 100operating in a teleoperation operating mode of operation withouttransitioning to an automated mode of operation. Referring still to FIG.23, the functions described with reference to the flow diagram of FIG.23 in one embodiment can be performed with system 100 transitioning oneor more times between an automated mode of operation and a teleoperationmode of operation.

With reference again to FIG. 3, FIG. 3 depicts system 100 to incorporateand use aspects described herein. System 100 includes robot 102, robotcontroller 150, operator computer system 170, and backend computersystem 160, all coupled via communication links 140 a-140 d. Thephysical locations of these components relative to one another can vary.For instance, they may be as close together as a few feet or as farapart as thousands of miles or more.

Communication links 140 a-140 d between the components may be realizedby any of various wireless and/or wired technologies (e.g.fiber-optic/radio/cable on different types and layers of dataprotocols). In some embodiments, one or more such communication linksincludes existing infrastructure, such as existing Ethernetinstallations operating over one or more local or wide area network(s).A non-limiting list of additional communication link technologiesincludes wireless-LAN (WLAN), Bluetooth, ZigBee, near-field, or otherwireless links, point-to-point radio systems or laser-optical systems,and satellite communication links, as examples.

Robot 102 may be any type of robot, such as an industrial robot offeredby ABB Inc. of Auburn Hills, Mich., U.S.A., as an example. Exemplaryrobots have several (usually 4, 5, 6 or 7) degrees of freedom enablingthem to perform any of various tasks usually characterized by themovement and/or manipulation of objects. In this regard, a robot refersin its broadest sense to an assembly that has multiple degrees offreedom.

Robot functions are served by different, and typicallyapplication-specific, components, some of which are depicted as part ofrobot 102 of FIG. 3. It should be understood that robot 102 includesadditional components omitted from FIG. 3 for convenience purposes, andfurther that a robot to incorporate/use aspects described herein neednot necessarily include each of the components depicted in FIG. 3.

Camera system 104 provides imaging functions enabling the robot to ‘see’physical objects in its environment. Camera system 104 may be realizedby proprietary and/or application-specific imaging device(s) orcommercial off-the-shelf (COTS) offerings providing 2-dimensional,3-dimensional, and/or depth-sensing imaging capabilities. An exampleCOTS product is the Kinect® motion controller offered by MicrosoftCorporation. Vision assistance system 106 is another type of sensorsystem providing vision capabilities for the robot. It can also includeimaging device(s) for imaging objects of the environment to assist inproximity or other spatial determinations. In some embodiments, thecamera system 104 is mounted on the robot at or near a ‘head’ thereofproviding a roughly first person perspective of the robot's activities,while the vision assistance system 106 includes multiple cameras mountedaround the robot 102 and imaging the robot 102 and its surroundings toprovide a third person perspective of robot activities with respect tothe surrounding objects.

Other sensor/sensor devices 108 are included to provide additionalsensing capabilities. The particular additional sensors may be dependenton the types of tasks the robot will perform when in operation. Anon-limiting list of additional sensors are microphones, positionsensors, proximity sensors, and force/pressure sensors, as examples.Some sensor devices can include data processing capabilities. So-calledsmart sensors are usually, though not always, connected directly torobot controller 150 via communication link(s). Other types of sensorsthat lack on-board data processing capability to process captured datamay provide data to a separate data processing device (such as robotcontroller 150, backend computer system 160, operator computer system170, and/or another computer system, not pictured) to process the sensordevice output.

Some sensors may be local to or remote from the robot 102 or robotcontroller 150. Remote sensor devices can provide input signals to therobot controller 150 that the robot controller 150 uses to control therobot 102 in performance of work described herein.

Some sensors that are mounted either on the robot 102 or at otherlocations can detect, or provide data that is processed to detect,obstacles that enter into or otherwise appear in the workspace of therobot 102. Sensor information including data about the detected obstaclecan be processed and used by robot controller 150 for position and othergeometric information. With respect to a smart sensor, a data processingcapability thereof can fully or partially process sensor information andtransfer to the robot controller 150 only the relevant data about theobstacle. In other types of sensors that lack data processingcapability, the sensor information can be processed by another dataprocessing device as described above and provided to robot controller150.

The robot 102 can also include a tool assembly 102T having actuator(s)or other devices (collectively referred to as actuator system 110)incorporated into, mounted to, or next to, the robot 102 to provideobject manipulation capabilities for manipulating or moving objects.Actuator system 110 may include grippers, claws, fixtures, suctiondevices, conveyors, twisting mechanisms, charging hose and/or wirefeeder mechanisms, and specialized equipment like medical tools, weldingguns, or spraying guns. Tool assembly 102T can be disposed at an end ofa robot arm defined e.g. by links 102L and joints 102J.

Robot controller 150 provides motion data to robot 102 to controlactions thereof. Motion data includes commands, as examples, sent to andreceived by component(s) of the robot that cause the components to driverobot actions, movement to other locations, and other activities.Accordingly, robot controller 150 may be a computer system havingprograms (i.e. instructions, program code) that execute to providemotion data to the robot 102 to control motion of the robot 102 toperform work. For instance, the robot 102 may hold a tool (not shown)used to perform work on a stationary or moving workpiece (also notshown), or may hold the workpiece to have work performed on it by anappropriate tool, as examples. As further examples, the robot 102 mayinclude a feeder component to feed a charging hose 202 or other objectinto drill holes 18 or other spaces to accomplish a task such asplanting explosive material. The feeder component may include a twistingmechanism configured to twist the charging hose 202 or other object inorder to more reliably reach desired locations.

Sensor information provided to the robot controller 150 directly orindirectly, such as via a data processing component to process sensordata, may be used to generate a safety zone in which the robot 102 maywork. When obstacles enter into the workspace of the robot 102, based onpositioning of the obstacle or the robot 102, a map can be built toinform the robot controller 150 of the safety zone. Additionally,information from sensor(s) or other components, such as camera system104, vision assistance system 106 and/or sensor/sensing device 108, canbe used by the robot controller to build a distance map and/or2/3-dimensional map. In some examples, raw sensor information isprocessed to build the map.

Robot controller 150 is in communication with operator computer system170 used in controlling and/or observing robot behavior. The operatorcomputer system 170 can show on a display thereof actual data aboutrobot motion and attached processes, for example, camera images,acoustic feedback, and sensor values. Additionally, operator computersystem 170 can act as a data processing device for sensor information,and can process data in both directions (i.e. data to/from the sensors).Operator computer system 170 may be implemented by any computer systemdesired, for instance an industrial personal computer or a programmablelogic controller, as examples.

Some embodiments may feature at least partial control of the robot 102by operator computer system 170. The operator computer system 170 canplay a role in dictating, selecting, building, and/or providing commandsor other signals to the robot 102 and/or robot controller 150 to causethe robot 102 to perform actions. In some examples, the operatorcomputer system 170 can include at least one actuator, such as joysticksor stylus-type devices that the operator can use to create continuousmotion signals (position and/or speed signals) that can be provided tothe robot 102 directly or via robot controller 150. Some actuators canprovide feedback to an operator based on, for example, input fromsensors of the robot 102. Feedback can be any kind of feedback that canbe sensed by an operator. An example is haptic or force feedback thatcauses a vibration in a joystick or a stylus. An actuator provided by ahaptic motion controller can provide tactile feedback to an operator. Inone embodiment, a haptic motion controller can be a PHANTOM OMNI® motioncontroller available from Sensable Technologies. In one embodiment, asset forth herein, system 100 can be operative to provide motion to oneor more charging component that corresponds to motion that is impartedto motion imparted by an operator to an actuator such as a haptic motioncontroller.

The operator computer system can also include a safety enable device,such as a three-position switch, to provide the ability for the operatorto enable or disable power to the robot 102 and/or other components ofthe system 100.

Backend computer system 160 can provide additional local or remotecomputing resources to support robot controller, operator computersystem 170, and/or robot 102. In this regard, control of the robot 102and/or other processes supporting robot tasks may be more demanding thancan be handled by the front-end systems. A local or remote backendfacility may be provided by backend computer system 160, and thefront-end components can off-load work to the backend computer system160. By way of specific example, processing of image data, especially3-dimensional image data, may present a significant burden on the robot102, sensors thereof, and/or robot controller 150. The image data may beprovided in part or whole to backend computer system 160 for processingand results can be provided back to the robot controller 150 or anothercomponent for use in robot processes.

As mentioned prior, components of system 100 need not be locatedadjacent to each other. Backend computer system 160 may be locatedon-site or offsite, for instance as a remote cloud-based computingfacility that offers a web-based data-processing solution.

One or more of the foregoing components of system 100 may be encompassedby, or included in, one or more other of the foregoing components.Similarly, functionality described above of a given component may beincorporated into a different component of the foregoing components.Backend compute resources provided by backend computer system 160, forexample, may be included in the robot controller 150 or operatorcomputer system 170, or vice versa. In some embodiments, functionalityof robot controller 150 and/or operator computer system 170 isincorporated into backend computer system 160.

Processes described herein may be performed by one or more computersystems or other processing devices, as provided, e.g. by robot 102,robot controller 150, backend computer system 160 and operator computersystem 170 as set forth herein. An example computer system toincorporate and use aspects described herein is depicted and describedwith reference to FIG. 24. Computer system 800 includes one or moreprocessors 802, memory 804, and one or more I/O devices 806, which maybe coupled to each other by busses and other electrical hardwareelements (not depicted). Processor(s) 802 include any appropriatehardware component(s) capable of implementing functions, for instanceexecuting instruction(s) (sometimes alternatively referred to as code,firmware and/or software) retrieved from memory 804. Execution of theinstructions causes the computer system 800 to perform processes,functions, or the like, such as those described herein supportingcontrol and/or operation of a robot, and described herein with referenceto the flow diagrams of FIGS. 12-19.

In some examples, aspects described herein are performed by a pluralityof homogenous or heterogeneous computer systems coordinated tocollectively perform processes, functions, or the like, such as thosedescribed herein supporting control and/or operation of a robot.

Memory 804 includes hardware components or other storage devices tostore data such as programs of instructions for execution, and otherdata. The storage devices may be magnetic, optical, and/orelectrical-based, as examples. Hard drives, field-programmable gatearrays (FPGAs), magnetic media, compact disks (CDs), digital versatiledisks (DVDs), and flash memories are example storage devices.Accordingly, memory 804 may be volatile, non-volatile, or a combinationof the two. As a specific example, memory 804 includes one or more harddrives and one or more random-access memory (RAM) devices for,respectively, non-volatile and volatile storage of data. Exampleprograms stored by memory include an operating system and applicationsthat run on the operating system, such as specialized applications toperform functions described herein. Memory 804 can store imagerepresentations obtained using camera system 104.

I/O device(s) 806 include hardware and/or software components thatsupport input and output of data to/from computer system 800. I/Odevice(s) 806 include physical components that attach physically orwirelessly to the computer system and/or integrate into the computersystem, such as keyboards, mice, display devices, joysticks, cameradevices, compact disks, thumb drives, printers, global positioningsystem (GPS) devices, gyroscopes, magnetometers, light sensors,proximity sensors, microphones, speakers, or accelerometers, asexamples. I/O devices 806 also include, but are not limited to, I/Ocontrollers and hardware and software supporting data communication withthe aforementioned components, such as network, graphics, and/or audiocontroller(s). An example I/O device 806 is a network adapter forcommunication of data between computer system 800 and another component,such as another computer system, across communication links. Examplesinclude Ethernet, cable, and/or fiber-based communications links passingdata packets between computer system 800 and other systems across one ormore networks, such as the Internet. Other example I/O devices 806include universal serial bus (USB), peripheral component interconnect(PCI), and serial adapters/interfaces configured to couple to devices oftheir respective kind.

A non-limiting list of example computer systems includes: personalcomputers (PCs), laptops, workstations, servers, mainframes, networkappliances, virtualization devices, computing terminals, personaldigital assistants, cellular telephones and smartphones, wearabledevices (“wearables”), tablet computers, and sensors such as cameras orcamera systems.

Accordingly, aspects described herein may take the form of one or moresystems, methods/processes, and/or a computer program products. Acomputer program product may be a computer-readable, tangible storagemedium or device that stores instructions. In some embodiments, thecomputer program product is non-transitory computer readable storagemedia. Referring to FIG. 25, an example computer program product 820 isdepicted that includes, for instance, one or more computer readablestorage media 824 to store computer-readable program code means, logicand/or instructions 828 thereon to provide and facilitate one or moreembodiments described herein.

A computer-readable storage medium can be, as examples, electronic,magnetic, electromagnetic, optical, and/or semi-conductor-based.Examples include but are not limited to: random access memory, read-onlymemory, computer disks, flash memory, and optical storage media likecompact disks (CDs) or digital versatile disks (DVDs). As specificallyused herein, computer-readable storage media does not per se consist oftransitory signals, such as radio waves or other propagating signals.

Program code contained or stored in/on a computer readable storagemedium can be obtained and executed by a computer system (computer,processing system, data processing system, etc. including a componentthereof) and/or other device to cause the computer system, componentthereof, and/or other device to behave/function in a particular manner.The program code can be transmitted using any appropriate medium,including (but not limited to) wireless, wireline, optical fiber, and/orradio-frequency. Program code for carrying out operations to perform,achieve, or facilitate aspects described herein may be written in one ormore programming languages. In some embodiments, the programminglanguage(s) include object-oriented and/or procedural programminglanguages such as C, C++, C#, Java, etc. Program code may executeentirely or partially on the computer system, a remote computer system,or a combination of partially on the computer system and partially on aremote computer system.

Program code can include program instructions obtained for execution byprocessor(s). Computer program instructions may be provided toprocessor(s) of, e.g., a computer system, to produce a machine, suchthat the program instructions, when executed by the processor(s),perform, achieve, or facilitate aspects described herein, such asactions, processes, or functions described in flowcharts and/or blockdiagrams described herein. Thus, each block, or combinations of blocks,of the flowchart illustrations and/or block diagrams depicted anddescribed herein can be implemented, in some embodiments, by computerprogram instructions. Behaviors/functions specified or performed by oneor more blocks may occur in a different order than depicted and/ordescribed, or may occur simultaneous to, or partially/wholly concurrentwith, one or more other blocks.

A system of one or more computers can be configured to performparticular operations or actions by virtue of having software, firmware,hardware, or a combination of them installed on the system that inoperation causes or cause the system to perform the actions. One or morecomputer programs can be configured to perform particular operations oractions by virtue of including instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the actions. Onegeneral aspect includes a method including: placing a robot at aposition in front of a mining wall, the robot having a tool assemblythat includes a camera system. The method also includes obtaining one ormore image representation using the camera system; positioning with therobot one or more charging component for entry into a drill hole, thepositioning including using image data obtained with the camera system.The method also includes moving with the robot the one or more chargingcomponent within the drill hole; and feeding with the robot explosivematerial into the drill hole. Other embodiments of this aspect includecorresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Implementations may include one or more of the following features. Themethod where the method includes registering detected drill holesdetected using the one or more image representation in a drilling map.The method where the obtaining includes performing a prescan of a walland determining a scan path based on image data obtained during theprescan, and where the obtaining further includes scanning a wall anddetecting drill holes represented in image data obtained during thescanning. The method where the method includes picking up one or morecomponent of the one or more charging component from a magazine. Themethod where the method includes activating the magazine to assemble aprimer to a detonator of a detonator package. The method where the robotholds a charging hose, where the method includes picking up a detonatorpackage with the robot, where the picking up includes disposing thecharging hose about a detonator package. The method where the robotholds a charging hose, where the method includes disposing the charginghose held by the robot about a detonator package so that the detonatorpackage is held by the charging hose, where the method includesinserting the charging hose and the detonator package held by thecharging hose into the drill hole. The method where the robot holds acharging hose, where the method includes disposing the charging hoseheld by the robot about a detonator package so that the detonatorpackage is held by the charging hose, where the method includesinserting the charging hose and the detonator package held by thecharging hose into the drill hole, where the method includes feedingcharging material through the charging hose into the drill hole whileretracting the charging hose from the drill hole with the robot. Themethod where the method includes determining whether all detected drillholes have been filled, and picking up a next detonator package if alldetected drill holes have not been filled. The method where the one ormore charging component is selected from the group including of acharging hose, a detonator, a primer, a signal wire, and explosivematerial. The method where the tool assembly includes a camera system,where the positioning includes determining based on one or more imagerepresentation obtained with the camera system whether the tool assemblyis properly positioned for entry into a drill hole and responsively to adetermination that the tool assembly is not properly positionedpositioning the tool assembly based on one or more operator input. Themethod where the method includes obtaining one or more imagerepresentation with a camera system, where the method includes detectinga drill hole representation of the drill hole in the one or more imagerepresentation, and registering the drill hole as a detected hole in adrilling map. The method where the method includes providing a drillingmap having information of a sequence of drill holes to charge and adetonator package associated to first and second of drill holes of thesequence of drill holes, and where the method includes using thedrilling map to select a proper detonator package for use in charging acurrent drill hole being subject to charging. The method where themoving with the robot the one or more component within the drill holeincludes moving the one or more component from an opening of the drillhole to a distal end of the drill hole in an automated operating mode,where movement of the one or more component is independent of anycurrent operator input with the automated operating mode active. Themethod where image data of the one or more image representation includes3D point cloud image data. The method where the tool assembly isoperative to rotate the charging hose back and forth about alongitudinal axis of the charging hose to overcome obstruction offeeding the charging hose in the drill hole. The method where theobtaining includes obtaining with the tool assembly the detonatorpackage in the hollow end of the charging hose from a magazinesupporting a plurality of detonator packages. The method where themagazine is operative for assembling a plurality of primers onto aplurality of detonators. The method where the detonator package includesa detonator, a primer, and a signal wire. The method where the toolassembly includes a 3D camera system. The method where the positioningis based on a drilling map, the drilling map indicating a location of aplurality of drill holes. The method where the tool assembly includes acamera system, where the positioning includes positioning a charginghose based on one or more image representation obtained with the camerasystem, and where obtaining the one or more image representationincludes merging together image data from a plurality imagerepresentations of a plurality of portions of a mining wall. The methodwhere the robot is operative to perform one or more obstructionavoidance routine selected from the group including a push forward,retract, rotate, and wiggle. The method further including selecting thedetonator package based on an operator input. The method where thecharging hose is configured to be friction fit about a primer of adetonator package. The robot system where the robot system is configuredto transition from the automated mode of operation to the teleoperationmode of operation in response to a sensed condition being sensed. Therobot system where the robot system is configured to transition from theautomated mode of operation to the teleoperation mode of operation basedon one or more operator input. The robot system where the one or morecharging component is one or more of a charging hose, a detonator, aprimer or a signal wire. The robot system where for a time that theautomated mode of operation is active the robot with the tool assemblyperforms the charging component positioning procedure independent of anycurrent operator input. The robot system where the charging componentpositioning procedure is a positioning procedure for moving a chargingcomponent from an opening of a drill hole through to a distal end of thedrill hole. The robot system where the charging component positioningprocedure is a positioning procedure for avoiding an obstruction withina drill hole. The robot system where the charging component positioningprocedure is a positioning procedure for positioning a chargingcomponent for entry into a drill hole. The robot system where the toolassembly includes a camera system, and where the robot systemdeactivates the automated mode of operation based on one or more imagerepresentation obtained with the camera system. The robot system wherethe robot system is configured to transition from the automated mode ofoperation to the teleoperation mode of operation in response to a sensedcondition being sensed, the sensed condition being the condition thatthe one or more charging component has not reached a distal end of adrill hole. The robot system where the tool assembly includes a camerasystem, where the robot system is configured to transition from theautomated mode of operation to the teleoperation mode of operation inresponse to a sensed condition being sensed, the sensed condition beingthe condition that the one or more charging component is not positionedproperly for drill hole entry, the sensed condition being determined byprocessing of one or more image representation obtained with the camerasystem. The robot system where the robot system in the teleoperationmode of operation activates a charging component positioning procedureselected from a group including (a) pushing in (b) retracting (c)rotating and (d) wiggling based on one or more operator input. The robotsystem where the robot system in the teleoperation mode of operationprovides motion to the one or more charging component that correspondsto motion that is imparted by an operator to an actuator of an operatorcomputer system. The robot system where the one or more operator inputis an input of a supervisor operator entered into a user interface of aremote operator computer system located remote from the robot. Themethod where the activating includes displaying on a display image dataof the one or more image representation. The method where the activatingincludes displaying on a display image data of the one or more imagerepresentation with a highlight selectable by an operator. The methodwhere the first one or more image representation and the second one ormore image representation include 3D point cloud image data. The methodwhere the out putted result includes an indication of a volume ofmaterial removed from the mining wall by the blasting. The robot systemwhere the tool assembly is operative to perform each of (a) pushing theone or more charging component forwardly, (b) retracting the one or morecharging component, and (c) rotating the one or more charging component.The robot system where the camera system is a 3D camera system for usein obtaining 3D point cloud image data. The robot system where the oneor more charging component includes a charging hose and where the robotis operative to feed explosive material through the charging hose whileretracting the charging hose. The robot system where the robot forperforming the one or more movement provides motion to the one or morecharging component that corresponds to motion currently being impartedto an actuator by an operator. The robot system where the robot isconfigured to operate in an automated mode of operation in which therobot with the tool assembly automatically performs a charging componentpositioning procedure for controlling a positioning of one or morecharging component, where the charging component positioning procedureis selected from the group including of (a) a procedure for moving theone or more charging component from an opening of a drill hole to thedistal end of the drill hole, (b) a procedure for positioning the one ormore charging component for drill hole entry, and (c) a procedure forpositioning the one or more charging component to avoid an obstructionwithin a drill hole. Implementations of the described techniques mayinclude hardware, a method or process, or computer software on acomputer-accessible medium.

One general aspect includes a method including: obtaining with a toolassembly of a robot a detonator package in a hollow end of a charginghose. The method also includes positioning with the tool assembly thecharging hose having the detonator package adjacent to an opening of adrill hole. The method also includes moving with the tool assembly thecharging hose having the detonator package along a length of the drillhole so that the detonator package and an end of the charging hose isdisposed at a distal end of the drill hole. The method also includesfeeding explosive material into the charging hose to deposit thedetonator package at the distal end of the drill hole and depositexplosive material along the length of the drill hole while removing thecharging hose from the drill hole. Other embodiments of this aspectinclude corresponding computer systems, apparatus, and computer programsrecorded on one or more computer storage devices, each configured toperform the actions of the methods.

Implementations may include one or more of the following features. Themethod where the tool assembly is operative to rotate the charging hoseback and forth about a longitudinal axis of the charging hose toovercome obstruction of feeding the charging hose in the drill hole. Themethod where the obtaining includes obtaining with the tool assembly thedetonator package in the hollow end of the charging hose from a magazinesupporting a plurality of detonator packages. The method where themagazine is operative for assembling a plurality of primers onto aplurality of detonators. The method where the detonator package includesa detonator, a primer, and a signal wire. The method where the toolassembly includes a 3D camera system. The method where the positioningis based on a drilling map, the drilling map indicating a location of aplurality of drill holes. The method where the tool assembly includes acamera system, where the positioning includes positioning a charginghose based on one or more image representation obtained with the camerasystem, and where obtaining the one or more image representationincludes merging together image data from a plurality imagerepresentations of a plurality of portions of the mining wall. Themethod where the robot is operative to perform one or more obstructionavoidance routine selected from the group including a push forward,retract, rotate, and wiggle. The method further including selecting thedetonator package based on an operator input. The method where thecharging hose is configured to be friction fit about a primer of adetonator package. The robot system where the robot system is configuredto transition from the automated mode of operation to the teleoperationmode of operation in response to a sensed condition being sensed. Therobot system where the robot system is configured to transition from theautomated mode of operation to the teleoperation mode of operation basedon one or more operator input. The robot system where the one or morecharging component is one or more of a charging hose, a detonator, aprimer or a signal wire. The robot system where for a time that theautomated mode of operation is active the robot with the tool assemblyperforms the charging component positioning procedure independent of anycurrent operator input. The robot system where the charging componentpositioning procedure is a positioning procedure for moving a chargingcomponent from an opening of a drill hole through to a distal end of thedrill hole. The robot system where the charging component positioningprocedure is a positioning procedure for avoiding an obstruction withina drill hole. The robot system where the charging component positioningprocedure is a positioning procedure for positioning a chargingcomponent for entry into a drill hole. The robot system where the toolassembly includes a camera system, and where the robot systemdeactivates the automated mode of operation based on one or more imagerepresentation obtained with the camera system. The robot system wherethe robot system is configured to transition from the automated mode ofoperation to the teleoperation mode of operation in response to a sensedcondition being sensed, the sensed condition being the condition thatthe one or more charging component has not reached a distal end of adrill hole. The robot system where the tool assembly includes a camerasystem, where the robot system is configured to transition from theautomated mode of operation to the teleoperation mode of operation inresponse to a sensed condition being sensed, the sensed condition beingthe condition that the one or more charging component is not positionedproperly for drill hole entry, the sensed condition being determined byprocessing of one or more image representation obtained with the camerasystem. The robot system where the robot system in the teleoperationmode of operation activates a charging component positioning procedureselected from a group including (a) pushing in (b) retracting (c)rotating and (d) wiggling based on one or more operator input. The robotsystem where the robot system in the teleoperation mode of operationprovides motion to the one or more charging component that correspondsto motion that is imparted by an operator to an actuator of an operatorcomputer system. The robot system where the one or more operator inputis an input of a supervisor operator entered into a user interface of aremote operator computer system located remote from the robot. Themethod where the activating includes displaying on a display image dataof the one or more image representation. The method where the activatingincludes displaying on a display image data of the one or more imagerepresentation with a highlight selectable by an operator. The methodwhere the first one or more image representation and the second one ormore image representation include 3D point cloud image data. The methodwhere the out putted result includes an indication of a volume ofmaterial removed from the mining wall by the blasting. The robot systemwhere the tool assembly is operative to perform each of (a) pushing theone or more charging component forwardly, (b) retracting the one or morecharging component, and (c) rotating the one or more charging component.The robot system where the camera system is a 3D camera system for usein obtaining 3D point cloud image data. The robot system where the oneor more charging component includes a charging hose and where the robotis operative to feed explosive material through the charging hose whileretracting the charging hose. The robot system where the robot forperforming the one or more movement provides motion to the one or morecharging component that corresponds to motion currently being impartedto an actuator by an operator. The robot system where the robot isconfigured to operate in an automated mode of operation in which therobot with the tool assembly automatically performs a charging componentpositioning procedure for controlling a positioning of one or morecharging component, where the charging component positioning procedureis selected from the group including of (a) a procedure for moving theone or more charging component from an opening of a drill hole to thedistal end of the drill hole, (b) a procedure for positioning the one ormore charging component for drill hole entry, and (c) a procedure forpositioning the one or more charging component to avoid an obstructionwithin a drill hole. Implementations of the described techniques mayinclude hardware, a method or process, or computer software on acomputer-accessible medium.

One general aspect includes a robot system including: a robot having atool assembly. The robot system also includes where the robot system isconfigured to operate in an automated mode of operation in which therobot with the tool assembly automatically performs a charging componentpositioning procedure for controlling a positioning of one or morecharging component. The robot system also includes where the robotsystem is configured to operate in a teleoperation mode of operation inwhich the robot with the tool assembly performs the charging componentpositioning procedure for controlling the position of the one or morecharging component based on one or more operator input. Otherembodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. Therobot system where the robot system is configured to transition from theautomated mode of operation to the teleoperation mode of operation inresponse to a sensed condition being sensed. The robot system where therobot system is configured to transition from the automated mode ofoperation to the teleoperation mode of operation based on one or moreoperator input. The robot system where the one or more chargingcomponent is one or more of a charging hose, a detonator, a primer or asignal wire. The robot system where for a time that the automated modeof operation is active the robot with the tool assembly performs thecharging component positioning procedure independent of any currentoperator input. The robot system where the charging componentpositioning procedure is a positioning procedure for moving a chargingcomponent from an opening of a drill hole through to a distal end of thedrill hole. The robot system where the charging component positioningprocedure is a positioning procedure for avoiding an obstruction withina drill hole. The robot system where the charging component positioningprocedure is a positioning procedure for positioning a chargingcomponent for entry into a drill hole. The robot system where the toolassembly includes a camera system, and where the robot systemdeactivates the automated mode of operation based on one or more imagerepresentation obtained with the camera system. The robot system wherethe robot system is configured to transition from the automated mode ofoperation to the teleoperation mode of operation in response to a sensedcondition being sensed, the sensed condition being the condition thatthe one or more charging component has not reached a distal end of adrill hole. The robot system where the tool assembly includes a camerasystem, where the robot system is configured to transition from theautomated mode of operation to the teleoperation mode of operation inresponse to a sensed condition being sensed, the sensed condition beingthe condition that the one or more charging component is not positionedproperly for drill hole entry, the sensed condition being determined byprocessing of one or more image representation obtained with the camerasystem. The robot system where the robot system in the teleoperationmode of operation activates a charging component positioning procedureselected from a group including (a) pushing in (b) retracting (c)rotating and (d) wiggling based on one or more operator input. The robotsystem where the robot system in the teleoperation mode of operationprovides motion to the one or more charging component that correspondsto motion that is imparted by an operator to an actuator of an operatorcomputer system. The robot system where the one or more operator inputis an input of a supervisor operator entered into a user interface of aremote operator computer system located remote from the robot. Themethod where the activating includes displaying on a display image dataof the one or more image representation. The method where the activatingincludes displaying on a display image data of the one or more imagerepresentation with a highlight selectable by an operator. The methodwhere the first one or more image representation and the second one ormore image representation include 3D point cloud image data. The methodwhere the out putted result includes an indication of a volume ofmaterial removed from the mining wall by the blasting. The robot systemwhere the tool assembly is operative to perform each of (a) pushing theone or more charging component forwardly, (b) retracting the one or morecharging component, and (c) rotating the one or more charging component.The robot system where the camera system is a 3D camera system for usein obtaining 3D point cloud image data. The robot system where the oneor more charging component includes a charging hose and where the robotis operative to feed explosive material through the charging hose whileretracting the charging hose. The robot system where the robot forperforming the one or more movement provides motion to the one or morecharging component that corresponds to motion currently being impartedto an actuator by an operator. The robot system where the robot isconfigured to operate in an automated mode of operation in which therobot with the tool assembly automatically performs a charging componentpositioning procedure for controlling a positioning of one or morecharging component, where the charging component positioning procedureis selected from the group including of (a) a procedure for moving theone or more charging component from an opening of a drill hole to thedistal end of the drill hole, (b) a procedure for positioning the one ormore charging component for drill hole entry, and (c) a procedure forpositioning the one or more charging component to avoid an obstructionwithin a drill hole. Implementations of the described techniques mayinclude hardware, a method or process, or computer software on acomputer-accessible medium.

One general aspect includes a method including: obtaining with a robotcamera system one or more image representation of a mining wall having aplurality of drill holes, the one or more image representation including3D point cloud image data. The method also includes comparing a drillingmap characterizing the mining wall to the one or more imagerepresentation to perform identification of a drill hole of the drillingmap not detected in the one or more image representation. The methodalso includes in response to the identification of a drill hole of thedrilling map not detected in the one or more image representation,activating operation where a robot system, based on one or more input ofan operator, designates image data of the one or more imagerepresentation as being representative of the drill hole. Otherembodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. Themethod where the activating includes displaying on a display image dataof the one or more image representation. The method where the activatingincludes displaying on a display image data of the one or more imagerepresentation with a highlight selectable by an operator. The methodwhere the first one or more image representation and the second one ormore image representation include 3D point cloud image data. The methodwhere the out putted result includes an indication of a volume ofmaterial removed from the mining wall by the blasting. The robot systemwhere the tool assembly is operative to perform each of (a) pushing theone or more charging component forwardly, (b) retracting the one or morecharging component, and (c) rotating the one or more charging component.The robot system where the camera system is a 3D camera system for usein obtaining 3D point cloud image data. The robot system where the oneor more charging component includes a charging hose and where the robotis operative to feed explosive material through the charging hose whileretracting the charging hose. The robot system where the robot forperforming the one or more movement provides motion to the one or morecharging component that corresponds to motion currently being impartedto an actuator by an operator. The robot system where the robot isconfigured to operate in an automated mode of operation in which therobot with the tool assembly automatically performs a charging componentpositioning procedure for controlling a positioning of one or morecharging component, where the charging component positioning procedureis selected from the group including of (a) a procedure for moving theone or more charging component from an opening of a drill hole to thedistal end of the drill hole, (b) a procedure for positioning the one ormore charging component for drill hole entry, and (c) a procedure forpositioning the one or more charging component to avoid an obstructionwithin a drill hole. Implementations of the described techniques mayinclude hardware, a method or process, or computer software on acomputer-accessible medium.

One general aspect includes a method including: obtaining with a robotcamera system a first one or more image representation of a mining wallin a state prior to blasting, the robot camera system being disposed onan arm of a robot. The method also includes obtaining with the robotcamera system a second one or more image representation of mining wallin a state subsequent to blasting. The method also includes comparingthe first one or more image representation to the second one or moreimage representation. The method also includes outputting a result ofthe comparing. Other embodiments of this aspect include correspondingcomputer systems, apparatus, and computer programs recorded on one ormore computer storage devices, each configured to perform the actions ofthe methods.

Implementations may include one or more of the following features. Themethod where the first one or more image representation and the secondone or more image representation include 3D point cloud image data. Themethod where the out putted result includes an indication of a volume ofmaterial removed from the mining wall by the blasting. The robot systemwhere the tool assembly is operative to perform each of (a) pushing theone or more charging component forwardly, (b) retracting the one or morecharging component, and (c) rotating the one or more charging component.The robot system where the camera system is a 3D camera system for usein obtaining 3D point cloud image data. The robot system where the oneor more charging component includes a charging hose and where the robotis operative to feed explosive material through the charging hose whileretracting the charging hose. The robot system where the robot forperforming the one or more movement provides motion to the one or morecharging component that corresponds to motion currently being impartedto an actuator by an operator. The robot system where the robot isconfigured to operate in an automated mode of operation in which therobot with the tool assembly automatically performs a charging componentpositioning procedure for controlling a positioning of one or morecharging component, where the charging component positioning procedureis selected from the group including of (a) a procedure for moving theone or more charging component from an opening of a drill hole to thedistal end of the drill hole, (b) a procedure for positioning the one ormore charging component for drill hole entry, and (c) a procedure forpositioning the one or more charging component to avoid an obstructionwithin a drill hole. Implementations of the described techniques mayinclude hardware, a method or process, or computer software on acomputer-accessible medium.

One general aspect includes a robot system including: a robot includinga robot arm, the robot arm having a plurality of links and a pluralityof joints. The robot system also includes a tool assembly disposed at adistal end of the robot arm, where the tool assembly has a camera systemfor use in obtaining image data, and where the tool assembly isconfigured to hold one or more charging component. The robot system alsoincludes where the tool assembly is operative to perform one or moremovement selected from a group including (a) pushing the one or morecharging component forwardly, (b) retracting the one or more chargingcomponent, and (c) rotating the one or more charging component. Otherembodiments of this aspect include corresponding computer systems,apparatus, and computer programs recorded on one or more computerstorage devices, each configured to perform the actions of the methods.

Implementations may include one or more of the following features. Therobot system where the tool assembly is operative to perform each of (a)pushing the one or more charging component forwardly, (b) retracting theone or more charging component, and (c) rotating the one or morecharging component. The robot system where the camera system is a 3Dcamera system for use in obtaining 3D point cloud image data. The robotsystem where the one or more charging component includes a charging hoseand where the robot is operative to feed explosive material through thecharging hose while retracting the charging hose. The robot system wherethe robot for performing the one or more movement provides motion to theone or more charging component that corresponds to motion currentlybeing imparted to an actuator by an operator. The robot system where therobot is configured to operate in an automated mode of operation inwhich the robot with the tool assembly automatically performs a chargingcomponent positioning procedure for controlling a positioning of one ormore charging component, where the charging component positioningprocedure is selected from the group including of (a) a procedure formoving the one or more charging component from an opening of a drillhole to the distal end of the drill hole, (b) a procedure forpositioning the one or more charging component for drill hole entry, and(c) a procedure for positioning the one or more charging component toavoid an obstruction within a drill hole. Implementations of thedescribed techniques may include hardware, a method or process, orcomputer software on a computer-accessible medium.

While the present application has been described with reference to anumber of specific embodiments, it will be understood that the truespirit and scope of the application should be determined only withrespect to claims that can be supported by the present specification.Further, while in numerous cases herein wherein systems and apparatusesand methods are described as having a certain number of elements it willbe understood that such systems, apparatuses and methods can bepracticed with fewer than or more than the mentioned certain number ofelements. Also, while a number of particular embodiments have been setforth, it will be understood that features and aspects that have beendescribed with reference to each particular embodiment can be used witheach remaining particularly set forth embodiment.

1. A method comprising: placing a robot at a position in front of amining wall, the robot having a tool assembly that includes a camerasystem; obtaining one or more image representation using the camerasystem; positioning with the robot one or more charging component forentry into a drill hole, the positioning including using image dataobtained with the camera system; moving with the robot the one or morecharging component within the drill hole; and feeding with the robotexplosive material into the drill hole.
 2. The method of claim 1,wherein the method includes registering detected drill holes detectedusing the one or more image representation in a drilling map.
 3. Themethod of claim 1, wherein the obtaining includes performing a prescanof a wall and determining a scan path based on image data obtainedduring the prescan, and wherein the obtaining further includes scanninga wall and detecting drill holes represented in image data obtainedduring the scanning.
 4. The method of claim 1, wherein the methodincludes picking up one or more component of the one or more chargingcomponent from a magazine.
 5. The method of claim 4, wherein the methodincludes activating the magazine to assemble a primer to a detonator ofa detonator package.
 6. The method of claim 1, wherein the one or morecomponent includes a charging hose, wherein the robot holds the charginghose, wherein the method includes picking up a detonator package withthe robot, wherein the picking up includes disposing the charging hoseabout a detonator package.
 7. The method of claim 1, wherein the one ormore component includes a charging hose, wherein the robot holds acharging hose, wherein the method includes disposing the charging hoseheld by the robot about a detonator package so that the detonatorpackage is held by the charging hose, wherein the method includesinserting the charging hose and the detonator package held by thecharging hose into the drill hole.
 8. The method of claim 1, wherein theone or more component includes a charging hose, wherein the robot holdsthe charging hose, wherein the method includes disposing the charginghose held by the robot about a detonator package so that the detonatorpackage is held by the charging hose, wherein the method includesinserting the charging hose and the detonator package held by thecharging hose into the drill hole, wherein the method includes feedingcharging material through the charging hose into the drill hole whileretracting the charging hose from the drill hole with the robot.
 9. Themethod of claim 1, wherein the method includes determining whether alldetected drill holes have been filled, and picking up a next detonatorpackage if all detected drill holes have not been filled.
 10. The methodof claim 1, wherein the one or more charging component is selected fromthe group consisting of a charging hose, a detonator, a primer, a signalwire, and explosive material.
 11. The method of claim 1, wherein thetool assembly includes a camera system, wherein the positioningcomprises determining based on one or more image representation obtainedwith the camera system whether the tool assembly is properly positionedfor entry into a drill hole and responsively to a determination that thetool assembly is not properly positioned positioning the tool assemblybased on one or more operator input.
 12. The method of claim 1, whereinthe method includes obtaining one or more image representation with acamera system, wherein the method includes detecting a drill holerepresentation of the drill hole in the one or more imagerepresentation, and registering the drill hole as a detected hole in adrilling map.
 13. The method of claim 1, wherein the method includesproviding a drilling map having information of a sequence of drill holesto charge and a detonator package associated to first and second ofdrill holes of the sequence of drill holes, and wherein the methodincludes using the drilling map to select a proper detonator package foruse in charging a current drill hole being subject to charging.
 14. Themethod of claim 1, wherein the moving with the robot the one or morecomponent within the drill hole includes moving the one or morecomponent from an opening of the drill hole to a distal end of the drillhole in an automated operating mode, wherein movement of the one or morecomponent is independent of any current operator input with theautomated operating mode active.
 15. The method of claim 1, whereinimage data of the one or more image representation includes 3D pointcloud image data.
 16. (canceled)
 17. (canceled)
 18. (canceled) 19.(canceled)
 20. (canceled)
 21. (canceled)
 22. (canceled)
 23. (canceled)24. (canceled)
 25. (canceled)
 26. (canceled)
 27. (canceled) 28.(canceled)
 29. (canceled)
 30. (canceled)
 31. (canceled)
 32. (canceled)33. (canceled)
 34. (canceled)
 35. (canceled)
 36. (canceled) 37.(canceled)
 38. (canceled)
 39. (canceled)
 40. (canceled)
 41. (canceled)42. (canceled)
 43. (canceled)
 44. (canceled)
 45. (canceled) 46.(canceled)
 47. A robot system comprising: a robot including a robot arm,the robot arm having a plurality of links and a plurality of joints; atool assembly disposed at a distal end of the robot arm, wherein thetool assembly has a camera system for use in obtaining image data, andwherein the tool assembly is configured to hold one or more chargingcomponent; and wherein the tool assembly is operative to perform one ormore movement selected from a group comprising (a) pushing the one ormore charging component forwardly, (b) retracting the one or morecharging component, and (c) rotating the one or more charging component.48. The robot system of claim 47, wherein the tool assembly is operativeto perform each of (a) pushing the one or more charging componentforwardly, (b) retracting the one or more charging component, and (c)rotating the one or more charging component.
 49. The robot system ofclaim 47, wherein the camera system is a 3D camera system for use inobtaining 3D point cloud image data.
 50. The robot system of claim 47,wherein the one or more charging component includes a charging hose andwherein the robot is operative to feed explosive material through thecharging hose while retracting the charging hose.
 51. The robot systemof claim 47, wherein the robot for performing the one or more movementprovides motion to the one or more charging component that correspondsto motion currently being imparted to an actuator by an operator. 52.The robot system of claim 47, wherein the robot is configured to operatein an automated mode of operation in which the robot with the toolassembly automatically performs a charging component positioningprocedure for controlling a positioning of one or more chargingcomponent, wherein the charging component positioning procedure isselected from the group consisting of (a) a procedure for moving the oneor more charging component from an opening of a drill hole to the distalend of the drill hole, (b) a procedure for positioning the one or morecharging component for drill hole entry, and (c) a procedure forpositioning the one or more charging component to avoid an obstructionwithin a drill hole.
 53. A method comprising: connecting a tool assemblyto a robot arm of a robot, the robot arm having a plurality of links anda plurality of joints; obtaining image data from a camera attached tothe robot arm; holding one or more charging components with the toolassembly; and moving the one or more charging components into a desiredlocation; wherein the moving includes moving the one or more chargingcomponents in a first direction within a drilled hole, retracting theone or more charging components in a direction opposite of the firstdirection, and/or rotating the one or more charging components.